^  AD- A  127  281 
UNCLASSIFIED 


THE  ROLE  OF  A  MENTAL  MODEL  IN  LEARNING  TO  OPERATE  A 
DEVICE (U)  ARIZONA  UNIV  TUCSON  DEPT  OF  PSYCHOLOGY 
D  E  KIERAS  ET  AL .  01  MAR  83  UARZ/DP/TR -83/ONR- 1 3 
N00014-81-C-0699  '  F/G  5/10 


END 


DT1C  FILE  COPY 


f 

C 

| 

1 


The  Role  of  a  Mental  Model  in 
Learning  to  Operate  a  Device 


David  E.  Kieras  and  Susan  Bovair 
University  of  Arizona 


Technical  Report  No.  13  (UARZ/DP/TR-83/ONR-13) 

March  1f  1983 


This  research  was  supported  by  the  Personnel  and  Training 
Research  Programs,  Office  of  Naval  Research,  under  Contract 
Number  N00014-81-C-0699,  Contract  Authority  Identification 
Number  NR  157-473.  Reproduction  in  whole  or  in  part  is 
permitted  for  any  purpose  of  the  United  States  Government. 


Approved  for  Public  Release;  Distribution  Unlimited. 


«) 


0 


tew-  -fcVWT- 


REPORT  DOCUMENTATION  PAGE 

READ  INSTRUCTIONS 

BEFORE  COMPLETING  FORM 

1  RECIPIENT'S  CATALOG  NUMBER 

No  *  13  ( UAKZ/mVTK-8  3/0NK- 13) 


4  TlTLt  (1*4  SubHilm) 

The  Hole  of  a  Mental  Model  in  Learning  to 
Operate  a  Device 


7  author/*; 

David  E.  Kieras  and  Susan  Bovair 


»  PERFORMING  ORGANIZATION  NAME  ANO  A00RES5 

Department  of  Psychology 
University  of  Arizona 

Tucson,  AZ  85721  _ _ 


it  controlling  office  name  and  adores* 

Personnel  and  Training  Research  Programs 
Office  of  Naval  Research  (Code  458) 

Arlington,  VA  22217  _ _ _ 

"TT  MONITORING'aGENCV  NA'mE  a  ADORESS/H  dllleient  tram  Controlling  Ottlce) 


<6  DISTRIBUTION  STATEMENT  (o I  Ihli  Report) 


S  TYPE  OF  REPORT  A  PERIOD  COVERED 

Technical  Report  1/1/81 


6  PERFORMING  org.  report  number 


«.  CONTRACT  OR  GRANT  NUMBER/ #7 

N00014-81-C-0699 


10  PROGRAM  ELEMENT.  PROJECT.  TASK 
AREA  *  WORK  UNIT  NUMBERS 

61153N;  RR  042-06 
RR  042-06-02;  NR  157-473 


12.  report  date 

March  1,  1983  _ 


U  NUMBER  OE  PAGES 

59 


IS.  SECURITY  CLASS.  fo»  (Mb  import) 

unclassified 


DECLASSIFICATION 'DOWN  GRADING 

SCHEDULE 


Approved  for  Public  Release;  Distribution  Unlimited 

17  DISTRIBUTION  STATEMENT  (ol  tho  obotroct  entered  In  Block  20,  II  boa.  Report) 


S.  KEY  WORDS  (Continue  on  rmvormo  oldo  It  MM«MW7  Ml d  Identity  by  block  ntmiber) 


Mental  models,  procedure  learning,  procedural  knowledge 


tvy  and  Identity  by  block  number) 


20  ABSTRACT  (Continue  on  rovoroo 


This  report  presents  two  studies  concerned  with  learning  how 
to  operate  a  simple  control  panel  device,  and  how  this  learning  is 
affected  by  understanding  the  internal  structure  of  the  device, 
which  is  a  device  model  for  the  device.  The  first  experiment 
compared  two  groups,  one  of  which  learned  a  set  of  operating 
procedures  for  the  device  by  rote,  and  the  other  learned  the 
device  model  before  receiving  the  identical  procedure  training. 
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The  model  group  learned  the  procedures  faster,  and  even  after  one 
week,  retained  them  better  and  executed  them  faster;  a  typical 
effect  size  is  a  20%  improvement.  Furthermore,  the  model  group 
could  simplify,  or  make  more  efficient,  the  propedyres  far  more 
often  then  the  rote  group.  The  second  study  examined  the 
hypothesis  that  the  improvement  is  due  to  the  model  group  being 
able  to  infer  the  procedures,  which  would  lead  to  more  rapid 
learning  and  better  recall  performance.  The  same  group 

manipulation  was  used,  but  subjects  inferred  the  procedures  rather 
than  learning  them,  and  "thought  out  loud"  while  doing  so.  The 
model  group  based  their  reasoning  on  direct  inferences  from  the 
device  model,  and  inferred  the  correct  procedures  in  almost  the 
minimum  amount  of  time.  The  rote  group  basically  followed  a 
trial-and-error  search  strategy,  and  arrived  at  similarly  correct 
procedures,  but  required  much  more  effort  than  the  model  group. 
Thus,  the  benefits  of  having  a  device  model  are  closely  related  to 
whether  it  supports  direct  and  simple  inference  of  the  steps  in 
the  operating  procedures. 
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Abstract 

yThis  report  presents  two  studies  concerned  with  learning  how 
to  operate  a  simple  control  panel  device,  and  how  this  learning  is 
affected  by  understanding  the  internal  structure  of  the  device, 
which  is  a  device  model  for  the  device.  The  first  experiment 
compared  two  groups,  one  of  which  learned  a  set  of  operating 
procedures  for  the  device  by  rote,  and  the  other  learned  the 
device  model  before  receiving  the  identical  procedure  training. 
The  model  group  learned  the  procedures  faster,  and  even  after  one 
week,  retained  them  better  and  executed  them'  faster;  a  typical 
effect  size  is  a  20S  improvement.  Furthermore,  the  model  group 
could  simplify,  or  make  more  efficient,  the  procedures  far  more 
often  then  the  rote  group.  The  second  study  examined  the 
hypothesis  that  the  improvement  is  due  to  the  model  group  being 
able  to  infer  the  procedures,  which  would  lead  to  more  rapid 
learning  and  better  recall  performance.  The  same  group 
manipulation  was  used,  but  subjects  inferred  the  procedures  rather 
than  learning  them,  and  "thought  out  loud"  while  doing  so.  The 
model  group  based  their  reasoning  on  direct  inferences  from  the 
device  model,  and  inferred  the  correct  procedures  in  almost  the 
minimum  amount  of  time. /V  The  rote  group  basically  followed  a 
tr ial-and-error  search  strategy,  and  arrived  at  similarly  correct 
procedures,  but  required  much  more  effort  than  the  model  group. 
Thus,  the  benefits  of  having  a  device  model  are  closely  related  to 
whether  it  supports  direct  and  simple  inference  of  the  steps  in 
the  operating  procedures. 
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The  Role  of  a  Mental  Model  in 
Learning  to  Operate  a  Device 


David  E.  Kieras  and  Susan  Bovair 
University  of  Arizona 

This  report  is  concerned  with  the  role  of  a  mental  model_  in 
learning  how  to  operate  an  unfamiliar  piece  of  equipment.  By 
"mental  model"  is  meant  some  kind  of  understanding  of  how  the 
piece  of  equipment,  or  device,  works  (cf.  Norman,  198?).  The 
work  in  this  report  is  concerned  with  having  people  acquire  the 
procedures  for  operating  a  simple  control  panel  device  consisting 
of  switches,  knobs,  lights,  and  push  buttons.  In  this  domain,  the 
mental  model  consists  of  how-it-works  knowledge  of  how  the  device 
works  inside.  Thus  a  mental  model  in  this  context  is  simply  the 
user’s  knowledge  of  the  internal  mechanisms  of  the  system.  In  the 
remainder  of  this  paper  this  type  of  mental  model  will  be  termed 
the  user  1 s  device  model . 

There  has  recently  been  considerable  work  on  the  nature  of 
mental  models  (see  Gentner  &  Stevens,  198?).  This  work  has  been 
mainly  concerned  with  extremely  complex  physical  systems,  and  with 
behavior  in  relatively  ill-defined  tasks,  such  as  describing  how 
an  electrical  circuit  works.  This  current  work  has  tended  to 
emphasize  mental  models  in  which  the  understanding  of  the  system 
approaches  the  level  of  understanding  that  an  actual  expert  in  the 
relevant  technical  field  would  have.  For  example,  considerable 
attention  has  been  devoted  to  how  people  acquire  a  fundamental 
understanding  of  electrical  circuits,  or  of  a  complex  shipboard 
steam  propulsion  system. 

There  are  in  fact  two  very  strong  intuitions  about  the  role 
of  a  device  model  in  learning  how  to  operate  a  device,  or  in  being 
able  to  operate  it  once  it  is  learned.  One  is  that  having  such 
knowledge  would  be  of  great  value;  the  other,  equally  strong, 
intuition,  is  that  it  is  not  likely  to  be  useful.  (See  Kieras  & 
Poison,  Notes  1,  2;  Bond  &  Towne ,  Note  3  for  more  discussion). 
The  actual  practice  of  the  technological  industries  suggest  very 
strongly  that  such  device  models  are  in  fact  unnecessary.  For 
example,  the  modern  telephone  system  is  in  fact  extremely  complex, 
but  very  few  people  actually  know  how  it  works,  even  those  who  are 
otherwise  highly  educated.  However,  almost  everybody  can 
successfully  operate  the  telephone  and  with  minimal  instruction 
can  even  place  elaborate  direct-dialed  overseas  calls.  Knowledge 
of  how  the  system  works  seems  to  be  essentially  irrelevant. 
Another  example  is  the  instructional  material  that  accompanies 
word  processing  systems.  Some  major  manufacturers  of  such  systems 
have  apparently  adopted  the  policy  that  instructional  material 
should  focus  just  on  how  to  get  the  job  done  with  the  system,  and 
should  not  contain  any  how-it-works  knowledge.  Apparently  this 
judgement  is  not  badly  wrong,  because  many  people  in  fact  learn 
how  to  operate  these  systems  successfully  within  a  reasonable 
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amount  of  time,  although  many  of  their  errors  and  difficulties  in 
doing  so  seem  to  be  related  to  the  lack  of  this  knowledge.  On  the 
other  hand,  it  appears  clear  that  people  can  infer  their  own 
device  models,  and  in  fact  they  do  display  some  how-it-works 
knowledge  for  familiar  devices  (Kieras,  Note  4).  One  would 
suppose  that  if  these  device  models  are  badly  inaccurate,  people 
will  perform  very  poorly  in  operating  the  device.  Thus,  the  role 
of  device  models  in  operating  equipment  remains  very  unclear, 
despite  these  strong  intuitions  and  the  importance  of  the 
practical  questions  involved. 

The  hypothesis  presented  here  is  that  having  a  model  of  a 
device  will  help  only  if  this  knowledge  makes  it  possible  to  infer 
the  procedures  for  operating  the  device.  This  knowledge  con  be 
very  superficial  and  yet  still  provide  this  function.  Thus,  in 
learning  procedures,  the  conceptual  depth  of  the  device  model  is 
nowhere  near  as  important  as  the  relevance  of  the  model  to  the 
inference  process.  This  marks  a  sharp  distinction  between  the 
present  approach,  and  the  work  currently  being  done  on  mental 
models,  which  tends  to  emphasize  the  depth  of  the  under  stand  ing 
rather  than  its  applicability,  or  relevance,  to  actual  task 
situations.  Thus,  the  investigation  of  the  effects  of 
how-it-works  knowledge  and  device  models  on  acquiring  procedural 
knowledge  has  to  be  sensitive  to  the  full  subtleties  of  how  such 
knowledge  relates  to  the  procedural  knowledge  being  learned. 

This  report  contains  two  studies.  The  first  is  a 
demonstration  that  providing  a  device  model  can  be  strongly 
facilitory.  The  second  demonstrates  that  the  device  model  can  be 
used  to  infer  procedures. 

EXPERIMENT  1 
Overview 

The  experiment  reported  here  was  fairly  complex.  In  outline, 
the  subjects  learned  a  set  of  procedures  for  operating  a  simple 
control  panel  device  consisting  of  switches  and  indicator  lights. 
The  goal  of  the  procedures  was  to  get  one  of  the  lights  to  flash. 
The  device  model  group  of  subjects  learned  some  how-it-works 
knowledge  in  the  form  of  a  description  of  the  device  based  on  the 
familiar  television  science  fiction  series  Star  Trek.  Namely, 
they  were  taught  that  the  device  was  the  control  panel  for  a 
"phaser  bank"  on  the  "Starship  Enterprise",  with  the  flashing 
light  indicating  a  successful  firing  of  the  phaser  bank.  Thus  the 
operating  procedures  could  be  explained  in  terms  of  the  mechanism 
of  this  fictitious  system.  The  rote  group  received  no  such 
instructions,  but  only  learned  the  procedures  "by  rote".  After 
learning  the  procedures,  both  groups  were  tested  immediately,  and 
after  one  week,  for  retention  of  the  procedures. 


The  experiment  was  designed  with  the  following  goals  in  mind: 
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1.  The  device  model  information  to  be  taught  to  subjects 
focussed  on  the  major  internal  components  of  the  system  and  their 
relationship  to  each  other,  and  to  the  controls  and  indicators. 
Essentially,  it  was  based  on  a  block  diagram  of  the  internal 
mechanisms  of  the  device,  and  this  diagram  was  also  provided  to 
the  subjects. 

2.  To  ensure  that  the  device  model  manipulation  was 
effective,  subjects  not  only  studied  the  model,  but  also  had  to 
successfully  pass  a  test  for  knowledge  of  the  model  before 
proceeding  to  the  procedure  training. 

3.  The  two  groups  received  exactly  the  same  procedure 
training;  this  was  made  possible  by  referring  to  the  device 
controls  with  abbreviations  that  could  be  used  for  both  groups. 

4.  Since  a  major  value  of  having  a  device  model  should  be  to 
facilitate  learning  and  using  the  device  in  unusual  situations, 
the  subjects  learned  procedures  to  apply  in  both  "normal"  and 
"malfunction"  situations. 

5.  Since  another  virtue  of  having  a  device  model  should  be 
the  ability  to  operate  a  device  more  efficiently,  some  of  the 
procedures  were  made  deliberately  inefficient,  and  the  subjects 
were  given  the  opportunity  to  devise  more  efficient  procedures. 

Method 

The  Device  and  Device  Model 


A  diagram  of  the  control  panel  used  for  the  device  is  shown 
in  Figure  1.  The  user  had  no  direct  knowledge  of  the  internal 
state  or  organization  of  the  device;  the  only  aspects  of  the 
device  that  the  user  was  directly  aware  of  was  the  settings  of  the 
switches,  and  whether  the  indicator  lights  were  on,  off,  or 
flashing.  The  simplest  way  to  explain  the  behavior  of  the  device 
is  to  present  the  actual  device  model  i.  was  presented  to  the 
subjects.  Figure  2  is  the  diagram  h  v  *3s  shown  to  the  subjects 
who  learned  the  device  model.  The  rsade?  should  refer  both  to 
Figure  1  and  to  Figure  2  while  reading  the  following  description. 
The  toggle  switch  labelled  SP  (Shipboard  Power)  is  the  on/off 
switch.  If  the  SP  switch  is  on,  the  pilot  light,  the  SP 
indicator,  lights.  The  power  flows  into  an  energy  booster  (FBI, 
which  if  operating  correctly  and  receiving  power,  lights  the  FP 
Indicator.  Power  flows  out  of  the  energy  booster  into  two 
accumulators,  labelled  MA  (main  accumulator)  and  SA  (secondary 
accumulator).  The  rotary  switch  ESS  (energy  source  selector) 
selects  which  accumulator  is  to  be  connected  to  the  PB  (phaser 
bank).  If  the  corresponding  pushbutton  is  pressed,  energy  flows 
from  the  accumulator  to  the  phaser  bank.  Notice  that  the  main 
accumulator  has  an  indicator  (MAI)  which  shows  that  the 
accumulator  is  receiving  energy  and  functioning  properly.  The 
secondary  accumulator  has  no  such  indicator;  consequently,  its 
internal  condition  is  not  indicated  to  the  user.  When  the  phaser 
bank  receives  the  energy,  the  indicator  labelled  PFI  (phaser 
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Figure  2.  The  block  diagram  representation  of  the  device  model 


firing  indicator)  flashes  4  times  at  roughly  half  second 
intervals,  and  then  stops. 

The  "standard  operating  procedure"  for  the  device  is  to  turn 
on  the  SP  switch,  select  either  the  MA  or  SA  with  the  ESS  knob, 
press  the  correspond ing  FM  or  FS  button,  wait  for  the  PFI  to 
finish  flashing,  then  return  the  ESS  to  the  N  position,  and  turn 
SP  off.  The  device  is  then  in  its  standard  initial  state. 

The  key  features  of  the  device  model  training  was  that  first 
of  all,  it  was  based  on  a  simple  description  of  the  major 
components  in  Figure  1,  along  with  a  description  of  how  these 
components  related  to  each  other,  and  how  the  controls  controlled 
the  flow  of  "energy"  from  one  component  to  another.  The  training 
did  not  attempt  to  explain  how  each  of  the  components  worked, 
except  to  point  out  that  they  sometimes  malfunctioned.  Appendix  A 
contains  the  exact  device  model  description  used,  with  the 
computer-assisted  instruction  details  edited  out  for  easy  reading. 
After  studying  the  description  of  the  device  model,  subjects  in 
the  device  model  group  were  given  a  multiple  choice  test  over  the 
major  concepts  that  the  materials  presented.  These  questions  were 
intended  to  determine  whether  the  subject  understood  the 
relationships  between  the  major  components  described  in  the  device 
model  description.  These  questions  are  presented  in  full  in 
Appendix  B. 

Apparatus .  The  device  itself  was  simulated  by  means  of  a 
laboratory  computer,  which  monitored  the  settings  of  the  switches 
and  push  buttons  and  controlled  the  indicator  lights  accordingly. 
All  instructions  and  commands  to  the  subjects  were  presented  on  a 
standard  video  terminal  positioned  next  to  the  device.  A 
computer-assisted  instruction  facility  was  used  to  present  all  of 
the  device  model  training,  the  procedure  training,  and  the  tests 
for  retention  of  procedures.  The  subject  was  seated  in  a  small 
room  at  a  table  with  the  terminal  and  the  control  panel,  and  were 
observed  by  means  of  a  closed  circuit  television. 

The  Operating  Procedures .  The  subjects  learned  two  kinds  of 
procedures  for  operating  the  device,  "normal"  and  "malfunction" 
procedures.  There  were  two  normal  procedures  for  operating  the 
device,  which  were  designated  simply  as  Procedure  _T_  and  Procedure 
2.  For  the  subjects  with  the  device  model,  these'  two  procedures 
corresponded  to  the  two  different  accumulators  of  the  phaser  bank 
system  that  could  be  used  to  supply  energy  to  fire  the  phasers. 
Of  course,  for  the  subjects  without  the  model,  these  two 
procedures  for  getting  the  PFI  light  to  flash  were  d istinqui shed 
only  by  the  different  control  actions  involved.  After  learning 
these  two  normal  procedures,  the  subjects  were  then  told  that 
sometimes  the  device  "malfunctioned",  and  when  this  happened,  it 
could  be  made  to  work  sometimes  by  following  an  alternative 
procedure.  These  mal function  situations  were  then  of  two  types; 
in  one  type,  the  PFI  could  be  made  to  flash  by  means  of  an 
alternate  procedure;  in  the  other  type,  the  PFI  could  not  be  made 
to  flash,  and  the  subject  was  to  follow  a  procedure  to  indicate 
that  an  irrecoverable  malfunction  had  occurred.  This  consisted  of 
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typing  the  letter  "E"  on  the  terminal  keyboard,  and  then  setting 
the  ES  selector  to  N  and  turning  the  SP  switch  off. 

For  the  subjects  with  the  device  model,  the  malfunction 
concept  was  explained  in  terms  of  the  idea  that  the  internal 
components  sometimes  failed,  but  since  there  was  some  redundancy 
in  the  system,  it  was  still  possible  to  fire  the  phasers  under- 
certain  conditions.  If  this  could  not  be  done,  the  Engineering 
section  of  the  starship  should  be  informed,  and  the  system  shut 
down.  The  subjects  who  did  not  have  the  device  model  were  told 
that  the  goal  was  to  get  the  PFI  light  to  flash),  and  that 
sometimes  the  device  did  not  work  correctly,  meaning  that. 
Procedure  1  and  Procedure  2  would  not  function  properly.  In  this 
case  they  were  to  use  an  alternate  procedure  to  get  the  PFI  to 
flash,  or  recognize  when  it  could  not  be  made  to  flash  and 
indicate  that  an  "Error  condition"  was  present.  For  each  normal 
procedure,  four  different  malfunctions  situations  were  defined, 
each  of  which  had  to  be  dealt  with  by  its  own  procedure.  This 
gave  a  total  of  10  procedures  for  the  subject  to  learn,  one  for 
each  of  the  10  situations,  both  normal  and  malfunction,  that  were 
used.  The  last  step  in  each  procedure  consisted  of  typing  the 
letter  "F"  for  "finished".  Tables  1  and  2  list  the  procedures  and 
situations,  which  are  described  in  terms  of  the  device  model. 

Some  of  the  procedures,  numbers  3,  5,  6,  and  7,  were  designed 
to  be  inefficient  so  that  subjects  could  improve  on  the  instructed 
procedures.  In  these  procedures,  the  subject  was  instructed  to 
follow  the  complete  normal  procedure  up  to  the  point  where  the  PF 
indicator  was  supposed  to  flash.  However,  it  was  possible  to  tell 
from  the  indicators,  as  soon  as  the  power  was  turned  on,  whether 
the  desired  procedure  would  work  at  all,  and  steps  could  then  be 
immediately  taken  to  either  try  the  other  accumulator,  or  to 
simply  shut  down  the  system.  By  including  such  procedures,  it  was 
possible  to  determine  whether  subjects  would  be  able  to  simplify, 
or  "short-cut",  the  instructed  procedures. 

Retention  Tests .  After  learning  all  10  procedures,  the 
subject  performed  three  retention  tests.  In  each  test,  a  command 
to  perform  either  Procedure  1  or  Procedure  2  would  appear  on  the 
subject’s  terminal,  and  he  or  she  would  then  attempt  to  perform 
the  designated  procedure.  On  half  of  the  trials,  a  normal 
situation  was  the  case,  so  the  normal  procedure  would  work;  on 
the  other  half,  there  was  some  malfunction  situation,  and  the 
subject  had  to  perform  the  appropriate  malfunction  procedure. 
There  were  10  different  situations,  corresponding  to  the  10 
different  procedures  that  were  trained.  No  feedback  was  given 
during  the  tests.  The  data  recorded  were  which  control  was 
operated  on  each  step  of  the  subject’s  procedure,  and  the 
corr espond ing  response  latency. 

A  total  of  16  trials  were  given  in  each  test,  half  of  them 
normal  situations,  half  malfunction  situations.  Situations  1  and 
2,  corresponding  to  Procedures  1  and  2,  were  each  presented  four 
times,  and  each  of  the  eight  malfunction  situations  once.  The 
order  of  presentation  of  the  different  situations  was  randomized 


Table  1 


Operating  Procedures  in  which 
Procedure  1  is  Commanded 


Situation : 

Normal 

Mai  function 

Procedure  No 
Status : 

. :  1 

All  OK 

3 

EB  out 

4 

PB  out 

5 

MA  out 

6 

MA  & 
out 

SA 

Steps 

1 

SP  on 

SP  on 

SP  on 

SP  on 

SP  on 

2 

ES  to  MA 

ES  to  MA 

ES  to  MA 

ES  to 

MA 

ES  to 

MA 

3 

press  FM 

press  FM 

press  FM 

press 

FM 

press 

FM 

4 

ES  to  N 

"E" 

"  E" 

ES  to 

SA 

ES  to 

N 

5 

SP  off 

ES  to  N 

ES  to  N 

press 

FS 

press 

FS 

6 

«p«i 

SP  off 

SP  off 

ES  to 

N 

"E” 

7 

itpti 

II  ptf 

SP  off 

ES  to 

N 

8 

ft  ptf 

SP  off 

9 

If  ptf 

Shortcut 

1 

SP  on 

SP  on 

SP 

2 

"E" 

ES  to  SA 

ES  ■  •  . 

3 

SP  off 

press  FS 

press  FS 

4 

If  pit 

ES  to  N 

"E" 

5 

SP  off 

ES  to  N 

6 

M  p»f 

SP  off 

7 

ft  pit 

8 

9 

Table  2 


Operating  Procedures  in  which 
Procedure  2  is  Commanded 


Situation:  Normal  Malfunction 


Procedure  No.:  278  9  10 

Status:  All  OK  EB  out  MA  &  SA  SA  out  SA  &  PB 

out  out  out 


Steps 


1 

2 

3 

4 

5 

6 

7 

8 

9 


Shortcut 


SP  on 
ES  to  SA 
press  FS 
ES  to  N 
SP  off 

It  ft! 


SP  on 
ES  to  SA 
press  FS 
"  E" 

ES  to  N 
SP  off 

n  pn 


SP  on 
ES  to  SA 
press  FS 
"E” 

ES  to  N 
SP  off 

it  pn 


SP  on 
ES  to  SA 
press  FS 
ES  to  MA 
press  FM 
ES  to  N 
SP  off 

it  pn 


SP  on 
ES  to  SA 
press  FS 
ES  to  MA 
press  FM 

it  pn 

ES  to  N 
SP  off 

n  pn 


1 

2 

3 


SP  on 

II  £11 

SP  off 
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for  each  subject  on  each  one  of  the  tests.  The  first  test  was 
given  immediately  after  completing  the  procedure  learning,  with 
subjects  being  instructed  to  perform  the  procedures  exactly  as 
they  had  learned  them.  After  completing  the  first  test,  the 
subject  took  the  same  test  again,  only  under  instructions  to  try 
to  short-cut  or  simplify  the  procedures  if  possible.  Following 
the  second  test,  the  subject  was  sent  home,  with  instructions  to 
return  in  one  week.  The  subject  then  took  the  very  same  test, 
with  no  particular  instructions  given  with  regard  to  short-cuts. 

Subjects .  Subjects  were  recruited  through  campus 
advertisements  and  they  were  paid  $5.00  for  each  session. 
Subjects  were  randomly  assigned  to  either  model  or  rote  group  so 
that  there  were  always  approximately  equal  numbers  of  subjects  in 
each  condition.  Due  to  apparent  sex  differences  noted  in  earlier 
work  (Kieras,  Note  3),  subjects  were  balanced  by  sex  so  that  there 
were  an  equal  number  of  males  and  females  in  each  condition.  Of 
the  40  subjects  22  were  males  and  18  were  females.  A  total  of  45 
subjects  were  run.  Two  subjects  quit  the  experiment  early  and 
their  partial  data  discarded.  Three  other  subjects'  data  was 
discarded  because  it  was  incomplete  due  to  data  recording 
problems.  Thus,  of  the  45  subjects,  a  total  of  40  yielded  data 
for  the  first  session,  with  20  in  each  group. 

Three  of  the  subjects  did  not  return  for  their  second 
session,  resulting  in  19  subjects  in  the  rote  condition  and  18  in 
the  model  condition.  A  one  week  delay  was  intended,  but  due  to 
scheduling  problems,  two  subjects  returned  ten  days  after  the 
first  session,  while  4  returned  eight  days  later. 

Procedure .  A  computer-assisted  instruction  program  on  the 
terminal  presented  the  experiment  instructions,  training  materials 
for  the  device  model,  and  training  on  the  procedures.  The  device 
model  group  and  the  rote  group  received  exactly  the  same  procedure 
training.  The  only  differences  in  what  the  two  groups  of  subjects 
saw  was  that  the  device  model  group  received  the  device  model 
instruction  at  the  beginning  of  the  experiment,  and  a  small  amount 
of  additional  device  model  instruction  before  starting  to  learn 
the  malfunction  procedures,  and  had  the  diagram  (Figure  1)  posted 
on  the  wall  in  front  of  them  throughout  training. 

Subjects  were  trained  on  the  procedures  in  an  order  slightly 
different  from  their  numbering;  the  two  normal  procedures,  1  and 
2,  were  trained  first,  followed  by  the  malfunction  procedures. 
Procedure  4  was  the  first  malfunction  procedure  trained,  followed 
by  Procedure  3,  then  Procedures  5  through  10.  The  process  of 
training  each  procedure  was  basically  a  serial  anticipation 
procedure,  alternating  with  a  test  procedure.  The  process 
consisted  of  a  teaching  pass  through  the  procedure,  followed  by  a 
trial  pass.  In  a  teaching  pass,  the  subject  was  prompted  to 
per  form  each  step,  and  could  try  operating  a  control,  or 
responding  with  a  "don't  know"  response.  If  the  response  was 
correct,  the  subject  was  told  so,  and  prompted  for  the  next  step. 
If  the  response  was  "don't  know"  or  incorrect,  the  subject  was 
informed  of  the  correct  step,  asked  to  carry  it  out,  and  then 
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prompted  for  the  next  step.  Alter  >er  forming  the  last  step  m  the 
teaching  pass,  the  subject  entered  ‘  ,■  ‘rial  pass,  and  was  asked 
to  try  executing  the  procedure  from  memory,  without  the  individual 
step  prompts.  If  the  entire  procedure  was  performed  correctly, 
another  trial  pass  was  prompted.  If  three  correct  trials  in  a  row 
were  performed,  tin?  subject  then  moved  on  to  the  next  procedure. 
If  an  incorrect  .  was  made,  the  subject  was  immediately 
informed,  and  then  the  teaching  pass  again. 

After  training,  the  di  .grim  was  removed  if  the  subjects  were 
in  the  model  condition,  and  the  subjects  performed  the  two 
retention  tests  described  above.  After  the  second  test,  subjects 
were  asked  to  describe  the  device  to  the  experimenter;  they  were 
asked  particularly  for  information  on  how  they  thought  the  device 
worked,  and  what  they  thought  the  controls  did.  One  week  later, 
subjects  returned  and  did  the  third  retention  test  described 
above.  After  they  had  completed  the  test,  they  were  once  again 
asked  to  describe  the  device. 


Results 


Device  Model  Training 

Subjects  in  the  model  group  took  an  average  of  1141  secs  to 
learn  the  device  model;  this  time  included  reading  the  training 
material  and  performing  the  quiz  to  the  criterion  of  all  questions 
answered  correctly.  Due  to  changes  in  the  programming,  specific 
information  on  which  quiz  questions  were  answered  correctly  is 
available  for  only  11  subjects.  The  errors  were  concentrated  on 
only  three  of  the  12  questions,  as  shown  in  Table  3-  The 
questions  about  the  function  of  the  ESS,  the  relation  of  FM  to 
ESS,  and  the  meanings  of  the  EB  and  MA  indicators  were  missed 
heavily.  This  suggests  that  the  relation  of  the  ESS  to  the  other 
components  and  the  nature  of  the  internal  states  of  the  system 
were  the  difficult  aspects  of  the  device  model. 

Procedure  Training 

Training  Time .  An  analysis  of  variance  was  performed  on  the 
total  training  time  for  each  procedure  in  each  group;  the  means 
are  shown  in  Table  4.  There  were  main  effects  of  group  and 
procedure,  and  an  interaction  between  group  and  procedure 
(jjs<.01).  The  main  result  is  that  subjects  in  the  model  group 
learned  a  procedure  faster  than  the  rote  subjects,  by  an  average 
of  76  seconds,  a  28%  improvement.  The  means  in  Table  4  were 
examined  with  a  Tukey  (a)  test  for  unconfounded  means  (Linton  & 
Gallo,  1975)  at  a  .05  significance  level  to  reveal  the  site  of  the 
effects . 

In  terms  of  the  relative  difficulty  of  the  procedures  within 
each  group,  an  interesting  pattern  appears.  For  the  rote  group, 
Procedures  1,  4,  5,  and  9  had  training  times  similar  to  each 
other,  and  significantly  longer  than  the  other  procedures,  which 
were  also  similar  to  each  other.  This  group  of  procedures  with 


Table  3 


Errors  made  on  Test  Questions 
during  Device  Model  Training  ( N  =  1 1 ) 


Number  of  subjects 

making  at  least 

one  error 

Quest i on 

What  does  ES  selector  do?  6 

What  does  it  mean  if  EB  10 

indicator  is  on? 

What  does  it  mean  if  MA  11 

indicator  is  on? 

No  malfunctions,  ES  to  7 

MA,  what  happens  if  press 

FM? 

What  does  SP  indicator  9 

ind icate? 

What  can  you  tell  from  1 2 

PF  indicator? 


TABLE  4 


Mean  Training  Times  (secs)  for  each 
Procedure  for  the  Two  Groups 


Procedure 

Model  Group 

Rote  Group 

1 

260  *  < 

427  * 

2 

138 

172 

3 

206 

197 

4 

374  * 

347  * 

5 

183  < 

390  « 

6 

165 

234 

7 

194 

173 

8 

121 

169 

9 

137  < 

380  * 

10 

159 

207 

Mean 

194 

270 

Notes.  * 

Procedure  took  longer  to  learn  than 
procedures  for  the  group.  Critical 
difference  is  164  sec. 

other 

value  for  the 

< 

The  model  group  was  significantly  faster  than  the  rote 
group  on  the  procedure. 

Table  5 

Mean  Errors  during  Teaching 

Phase  of  Training 

Procedure 

Model  Group 

Rote  Group 

1 

.59 

1 .06 

2 

.12 

.  12 

3 

.29 

.29 

4 

1.18 

.65 

5 

.41  < 

2.47  * 

6 

.29 

.59 

7 

.47 

.47 

8 

.53 

.41 

9 

.06  < 

2.59  # 

10 

.41 

.65 

Mean 

.44 

.93 

Notes . 

<  Groups  differed  significantly  on  this  procedure. 

*  Procedure  took  longer  to  learn  than  other  procedure 
for  the  group. 

Critical  value  for  difference  is  1.36  errors. 

Page  10 


relatively  long  times,  in  this  sense,  is  indicated  with  the 
asterisks  in  Table  4.  For  the  model  group,  Procedures  1  and  4 
have  similar  and  longer  times  compared  to  the  other  procedures. 
Thus,  in  terms  of  training,  the  very  first  procedure  (1)  and  the 
first  malfunction  procedure  (4)  are  relatively  hard  for  both 
groups.  But,  as  shown  in  the  Table,  the  model  group  has  shorter 
training  time  than  the  rote  group  for  all  but  two  of  the 
procedures,  but  are  significantly  different  only  for  Procedures  1, 
5  and  9.  In  these  cases  the  difference  is  substantial.  The  rote 
group  finds  Procedure  1  very  hard  compared  to  the  model  group, 
taking  64%  longer  to  learn.  Procedures  5  and  9,  which  both  deal 
with  first  exposures  to  an  accumulator  malfunction,  are  harder  by 
factors  of  about  2  and  3  for  the  rote  group  compared  to  the  model 
group . 


Errors .  The  mean  number  of  errors  made  during  the  teaching 
phase  of  training  are  shown  in  Table  5.  This  variable  shows 
similar  effects  to  these  found  for  the  training  time.  Because 
errors  made  in  the  teaching  phase  do  not  require  that  the  subject 
must  go  back  through  the  whole  teaching  again  (unlike  errors 
during  the  trial  phase),  there  would  not  necessarily  be  a  strong 
relationship  between  teaching  errors  and  training  time.  However, 
there  is  a  correlation  of  .50  between  these  two  variables  overall, 
and  similar  main  effects  of  group  (£<.05),  procedure  (£<.01),  and 
the  group  by  procedure  interaction  (p<.01).  As  shown  in  the 
Table,  rote  subjects  make  about  twice  as  many  errors  during  the 
teaching  as  model  subjects. 

Comparing  the  means  with  Tukey's  (a)  test  for  unconfounded 
means  as  before  showed  that  the  pattern  of  errors  is  quite 
different  for  the  two  groups.  In  the  model  group,  only  one 
procedure,  number  4,  produced  more  errors  then  the  other 
procedures,  but  not  significantly  so.  For  the  rote  group, 
Procedures  5  and  9  produce  a  similarly  large  number  of  errors, 
much  more  than  the  model  group.  Thus,  Procedures  5  and  9,  which 
deal  with  accumulator  malfunctions  for  each  of  the  two  normal 
procedures,  are  especially  difficult  to  master,  both  in  terms  of 
time  and  in  eliminating  errors. 

" Don ’ t  know”  Responses .  The  mean  number  of  "don't  know" 
responses  on  each  procedure  was  analyzed,  but  is  not  shown. 
Analysis  of  variance  shows  a  main  effect  of  procedure  ( p< .01 )  and 
a  group  by  procedure  interaction  (£=.05),  but  no  effect  of  group. 
The  number  of  "don't  knows"  drops  rapidly  over  the  first  four 
procedures  and  remains  low  thereafter.  This  is  a  weak  variable, 
since  the  use  of  this  response  by  the  subject  is  completely 
optional,  and  it  was  rarely  used  after  the  first  few  procedures, 
giving  essentially  identical  rates  for  almost  all  procedures. 
However,  it  is  a  relatively  direct  indicator  of  how  willing  the 
subject  is  to  guess  the  steps  of  a  procedure.  Applying  Tukey  (a) 
for  unfounded  means  gives  a  significant  difference  between  the  two 
groups  on  Procedure  2,  where  there  are  40%  more  "don't  knows"  for 
the  rote  subjects.  Using  simple  t-tests  for  more  power,  the  model 
group  also  produces  fewer  "don't  know"  responses  on  Procedures  1 
and  9  as  well.  These  results  imply  that  the  model  subjects  can 
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guess  more  of  the  procedure  steps  than  rote  subjects.  This 
appears  in  both  the  first  and  second  procedures,  and  in  one  of  the 
most  difficult  abnormal  procedures  as  well. 

Summary .  During  procedure  training,  model  subjects  were 
faster,  made  fewer  errors,  and  guessed  more  steps  than  the  rote 
subjects.  Over  the  entire  training  on  the  10  procedures,  model 
subjects  were  an  average  of  about  760  seconds  faster  than  rote 
subjects.  This  can  be  compared  with  the  average  1141  seconds  that 
it  took  subjects  to  learn  the  device  model  to  criterion. 

For  both  groups,  Procedure  1  takes  a  relatively  long  time  to 
learn,  but  rote  subjects  take  longer.  Thus  while  the  first 
procedure  is  relatively  difficult,  it  is  far  less  so  for  the  model 
subjects.  For  both  groups,  Procedure  2  is  rapidly  learned, 
reflecting  its  similarity  to  Procedure  1.  Procedure  5  is  learned 
rapidly  by  model  subjects  and  with  relatively  few  errors. 
However,  the  rote  subjects  find  this  procedure  much  more 
difficult,  both  taking  longer  and  making  more  errors,  than  model 
subjects.  Procedure  9  produces  a  similar  result.  Procedures  5 
and  9  both  involve  using  both  accumulators,  and  are  the  first  such 
for  each  of  the  two  normal  procedures.  For  the  rote  subjects, 
these  procedures  are  especially  difficult,  while  for  model 
subjects,  they  are  quite  str aight for ward .  Procedures  3,  4,  6,  7, 
8,  and  10  do  not  differ  for  the  two  groups. 

Retention  Tests 

A  detailed  analysis  of  the  retention  test  responses  was  done 
by  examining  the  pattern  of  individual  steps  produced  on  each  test 
trial  by  individual  subjects.  The  number  of  subjects  using  a 
particular  pattern  in  each  of  the  10  situations  was  tallied  for 
each  of  the  3  tests.  For  each  situation,  the  patterns  were 
classified  into  a  set  of  categories  and  the  frequencies  of  each 
category,  for  each  situation,  on  each  test,  were  tallied.  It  must 
be  kept  in  mind  that  since  some  subjects  failed  to  return  for  Test 
3,  and  there  were  an  unequal  number  who  did  so  in  the  two  groups, 
weighted  means  or  proportions  must  be  taken  over  the  three  tests. 

The  categories  developed  can  be  broadly  divided  into  2 
groups:  correct  and  incorrect .  The  correct  category  can  be 
further  subdivided  into  instructed  and  shor t-cut  procedures.  A 
response  pattern  was  called  a  corr ec t  instructed  proced ure  the 
subject  used  the  exact  procedure  that  had  been  trained  for  the 
situation.  A  short-cut  procedure  achieved  the  correct  final 
outcome  on  the  device,  but  excluded  some  of  the  unnecessary  steps 
that  were  part  of  some  of  the  instructed  procedures.  The  optimum 
shortcuts  procedures  are  shown  in  Tables  1  and  2;  notice  that 
these  shortcuts  exist  for  only  four  situations  (see  Tables  1  and 
2).  A  response  pattern  was  considered  incorrect  if  it  did  not 
achieve  the  correct  final  outcome  on  the  device,  or  involved 
additional,  unnecessary  steps  beyond  those  originally  instructed. 
About  1%  of  the  patterns  could  not  be  classified  due  to  data 
recording  errors.  For  brevity,  the  presentation  of  these  results 
is  limited  to  the  overall  results  for  the  three  tests  collapsed 
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over  situations.  The  results  for  individual  situations  are  shown 
only  for  the  third  test. 

Retention  accuracy.  The  proportions  of  correct  instructed 
and  short-cut  procedures  in  each  retention  test  are  shown  in  Table 

6.  Considering  the  instructed  and  short-cut  procedures  together 
as  correct  procedures,  Chi-square  tests  comparing  the 
distributions  of  correct  and  incorrect  responses  were  significant 
(£<  .03)  for  each  test  and  for  the  overall  distributions.  Model 
subjects  used  a  correct  procedure  in  a  situation  more  often  than 
rote  subjects  (80%  vs  67%,  X*2( 1 ) =41 . 9 ,  p< .01 ) .  This  difference 
can  also  be  seen  on  the  individual  retention  tests,  but  the 
greatest  difference  is  on  the  second  test.  Here,  when  subjects 
are  asked  to  try  to  short-cut  the  procedures,  the  model  subjects 
use  the  instructed  procedure  less  often  than  on  the  first  test, 
but  use  short-cuts  frequently,  giving  80%  correct  responses. 
However,  the  rote  subjects  are  correct  61%  of  the  time,  less  often 
than  on  the  first  test.  It  seems  that  rote  subjects  are  confused 
by  attempting  to  short-cut,  because  their  performance  on  the  third 
test,  where  there  was  no  request  for  short-cuts,  is  the  same  as  on 
the  first  test.  Model  subjects  perform  about  the  same  on  all 
three  tests,  with  a  slight  decline  by  the  third  test. 

Model  subjects  are  not  only  more  likely  to  use  a  correct 
procedure,  but  they  are  also  more  likely  to  use  a  short-cut. 
Overall,  model  subjects  short-cut  10%  of  the  time,  out  of  a 
possible  25%  of  the  retention  test  trials,  while  rote  subjects  did 
so  only  2%  of  the  time.  However,  the  better  performance  of  model 
subjects  is  not  only  due  to  their  more  frequent  use  of  short-cuts, 
but  also  because  they  use  the  instructed  procedure  more  often  than 
rote  subjects. 

The  long-term  retention  as  measured  by  Test  3  is  especially 
important.  The  model  subjects  show  a  somewhat  better  ability  to 
perform  correctly,  than  do  the  rote  subjects  (.79  vs.  .71,  11% 
improvement).  More  detail  on  Test  3  performance  appears  in  Table 

7,  which  shows  the  proportions  of  response  patterns  classified  as 
instructed,  short-cut,  and  total  correct  for  each  procedure 
situation  on  Test  3- 

Two  key  points  are  that  first,  after  one  week,  the  model 
subjects  car.  short-cut  a  procedure  much  more  often  than  the  rote 
subjects;  from  Table  6,  this  is  .12  versus  .04.  The  model 
subjects  were  short-cutting  48%  of  the  time  when  it  was  possible, 
and  the  rote  subjects  only  16%  of  the  time  when  it  was  possible. 
Note  especially  Situations  3  and  7,  in  Table  7,  in  which  the  EB 
being  out  implies  an  obvious  and  drastic  shortcut.  While  there  is 
little  difference  in  the  proportion  of  total  correct  responses, 
the  model  subjects  short-cut  about  twice  as  often  on  this 
procedure  as  the  rote  subjects.  The  second  point  is  that  the 
model  group  is  correct  more  often  than  the  rote  group  on  most 
procedures,  but  significantly  so  only  on  a  few.  Thus,  the  model 
helps  very  little  in  some  situations  (e.  g.,  Situation  1),  and  in 
other  situations,  the  improvement  is  substantial  (e.  g.  Situation 
9) . 
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Table  6 


Proportions  of  Retention  Test  Response 
Patterns  in  each  Category  for  each  group 


Category: 

Instructed 
.  Procedure 

Short-cut 

Procedure 

Total 

Correc  t 

Rote  Group 

(N  =  994) 

Tist  1 

.70 

0 

.70 

Test  2 

.57 

.04 

.60 

Test  3 

.67 

.04 

.71 

Overall 

.65 

.02 

.67 

Model  Group 

(N=928) 

Test  1 

.79 

.03 

.82 

Test  2 

.65 

.  15 

.81 

Test  3 

.67 

.12 

.78 

Overall 

.70 

.  10 

.80 

Table  7 

Proportions  of  Test  3  Responses 
in  each  Situation  in  each  Category 

Category 

Situations 

1 

2 

3 

4  5 

6 

7 

8 

9 

10 

Instr  ucted 

Procedure 

Rote 

.95 

.87 

.42 

.21  .53  • 

37 

.32 

.37 

.63 

.63 

Model 

.99 

.97 

.06 

.11  .17  . 

17 

.06 

.44 

1.00 

.83 

Short-Cuts 

Rote 

• 

.32 

.0 

05 

.21 

M. 

_ 

Model 

— 

.72 

.28  . 

33 

.56 

— 

Total  Correct 

Rote 

.95 

.87 

.74 

.21  .53  • 

42 

.53 

.37 

.63 

.63 

Model 

.99 

.97* 

.78* 

.11  .44  . 

50 

.61 

.44 

1.00* 

.83 

*  Group  difference  on  situation  is  significant  by  chi-square  test 
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Execution  Times .  The  time  to  execute  a  procedure  was 
measured  from  the  appearance  of  the  command  on  the  terminal  to  the 
typing  of  the  final  "F"  response.  Table  8  presents  the  mean  times 
for  the  three  tests,  averaged  over  all  procedures,  classified  by 
whether  the  response  pattern  was  a  instructed,  shortcut,  or  an 
incorrect  procedure.  Due  to  the  substantially  unequal  sample 
sizes  and  unequal  representation  of  individual  subjects  due  to 
errors  and  optional  use  of  shortcuts,  and  the  empty  cell  for  Test 
1  short-cuts,  a  simple  and  conservative  way  to  compare  these  times 
between  the  two  groups  was  with  multiple  independent-sample 
t-tests,  using  a  Bon ferronni-inequal ity  significance  level  of  .006 
for  each  of  the  8  comparisons.  Under  this  criterion,  all  of  the 
means  are  significantly  different  between  the  two  groups,  except 
for  the  Test  2  short-cut  times.  Thus,  the  model  subjects  were 
able  to  execute  the  instructed  procedures  faster,  perform 
short-cuts  faster  after  one  week,  and  even  perform  incorrect 
procedures  faster  as  well.  The  typical  improvement  is  about  20%. 
Notice  that  if  the  distinction  between  instructed  and  short-cut 
procedures  is  ignored,  the  advantage  of  the  model  group  is 
substantial,  (about  3.2  secs,  or  a  17%  improvement)  since  not  only 
are  they  generally  faster  than  the  rote  group,  but  also  use  the 
quicker  short-cut  procedures  more  often. 

Sources  of  Errors .  The  nature  of  the  errors  made  on  the 
retention  tests  was  examined  using  the  fully  detailed  response 
pattern  classification  collapsed  over  the  three  retention  tests. 
Of  the  total  of  1872  responses,  approximately  26%,  or  480,  were 
classified  as  errors.  These  error  frequencies  were  tabulated  into 
a  table  classified  by  group,  situation,  and  type  of  error.  In 
interpreting  the  results,  it  should  be  kept  in  mind  that  if  the 
subject  made  more  than  error  on  a  trial,  the  pattern  was 
classified  only  by  the  first  error  made  on  that  trial. 
Consequently  the  two  final  categories  in  the  table  have  the 
property  that  these  types  of  errors  would  have  been  made  later  in 
the  responses  classified  under  earlier  errors.  The  table  was 
subjected  to  a  three-factor  log-linear  analysis,  which  showed  that 
all  main  effects  and  two-factor  interactions  were  significant 
(jp<.01  ).  However,  the  three-factor  interaction  was  not 
significant.  This  means  that  the  distribution  of  types  of  errors 
differed  for  the  two  groups,  and  the  two  groups  differed-  in  the 
number  of  errors  made  in  the  different  situations,  and  the 
situations  differed  in  the  types  of  errors  produced.  However,  any 
one  of  these  two-factor  interactions  does  not  have  to  be  qualified 
by  the  level  of  the  third  factor.  As  already  described,  the  model 
group  produced  many  fewer  errors  than  the  rote  group,  and  clearly 
the  situations  differed  in  the  number  of  errors  they  produced,  and 
clearly  some  error  types  were  more  common  than  others.  The 
interaction  of  group  with  situation  on  errors  is  redundant  with 
the  similar  information  presented  above  for  correct  responses. 
The  interaction  of  procedure  with  type  of  error  is  basically 
uninteresting,  because  clearly  the  different  situations  would 
generate  different  error  types.  Thus,  of  these  three  two-factor 
interactions  and  the  main  effects,  only  the  two-factor  interaction 
of  group  with  type  of  error  will  be  discussed.  Table  9,  which  is 
collapsed  over  situations,  and  shows  the  percentage  of  errors 


Table  8 


Mean  Procedure  Execution  Times  (secs)  for 
each  Category  of  Response  Pattern 


Instructed 

Short-cut 

Incor r ec  t 

Procedure 

Procedure 

Rote  Group 

Test  1 

21.6 

29.2 

Test  2 

19.6 

15.9 

24.1 

Test  3 

19.2 

19.0 

24.5 

Means 

20.  1 

17.4 

26.0 

Model  Group 
TestT  1 - 

18.5 

... 

22.0 

Test  2 

15.6 

15.2 

16.5 

Test  3 

16.2 

14.0 

19.4 

Means 

16.8 

14.6 

19.2 

TABLE  9 


Proportion 

Both 

of  Errors  of 
Groups,  and 

Each  Type 
Overall . 

for 

TYPE 

Rote 

(N=300) 

Model 
( N= 1 74 ) 

Over 

Minor 

.14 

.  17 

.  15 

Other 

.05 

.01 

.04 

Wrong  Button 

.03 

.01 

.02 

Both  Accs. 

•  36 

•  38 

.37 

Fail  to  try  other 
Acc  . 

.20 

.  12 

.  17 

Fail  to  push  first 
button 

.05 

.28 

.13 

Push  button  on 
wrong  Acc . 

.  1 1 

.02 

.08 

Type  "E”  when 
shouldn ' t 

.04 

.00 

.02 

Fail  to  type  " E" 

.04 

.01 

.03 

Totals 


1.00 


1.00 


1.00 


produced  by  each  group,  and  over  both  groups,  for  each  of  the 
error  types  classified. 

About  15%  of  the  errors  were  minor  errors.  Such  errors  were 
tapping  the  space  bar  before  the  final  "F",  moving  the  ES  selector 
to  the  wrong  position  but  moving  it  to  the  correct  position  before 
firing,  typing  "E"  one  or  two  steps  late,  and  pressing  the  firing 
button  twice.  About  4 %  of  the  errors  do  not  classify  into  the 
other  categories. 

The  bulk  of  the  errors  are  produced  in  procedures  used  in  the 
malfunction  situations;  these  are  as  follows:  (1)  using  both 
accumulators  where  only  one  should  be  used;  (2)  failing  to  use 
both  accumulators  when  they  should  be;  (3)  failing  to  push  the 
firing  button  (either  FM  or  FS);  which  means  that  the  subject 
wrongly  concludes  that  it  is  not  possible  to  get  the  device  to 
operate,  and  so  will  then  later  type  in  the  "E"  response  showing 
that  the  device  is  inoperative;  (4)  pushing  the  button  with  the 
ESS  on  the  wrong  setting. 

These  four  major  error  types  show  some  interesting 
differences,  or  lack  of  differences,  between  the  two  groups.  The 
error  of  trying  both  accumulators  when  only  one  should  be  used 
appears  proportionately  equally  often  in  both  groups.  The  rote 
group  fails  to  try  the  other  accumulator  more  often  than  the  model 
group  does,  but  the  model  group  fails  to  push  a  button  (try  to 
fire)  far  more  often  than  the  rote  group  does.  Finally  the  rote 
group  is  more  likely  to  make  a  seriously  wrong  error  of  not  having 
the  accumulator  properly  selected  with  the  ESS. 

The  most  frequent  error  of  trying  both  accumulators 
improperly  is  a  very  common  pattern  and  is  related  to  the  fact 
that  Situation  4  was  done  correctly  only  20%  or  less  of  the  time. 
The  reason  for  Situation  4  having  such  low  accuracy  is  that  most 
of  the  subjects  (90%  of  rote  subjects,  80%  of  model  subjects) 
first  tried  to  use  the  main  accumulator  as  commanded,  but  then 
tried  using  the  secondary  accumulator  as  well.  However,  since  the 
main  accumulator  indicator  is  on  in  this  situation,  the  failure  of 
the  system  to  work  means  that  tHe  phasers  are  defective.  Thus, 
there  is  no  point  in  trying  the  secondary  accumulator.  It  appears 
that  subjects  generalized  the  various  malfunction  procedures  into 
one  that  simply  involved  trying  the  commanded  accumulator  first, 
and  then  always  trying  the  other  accumulator  before  concluding 
that  the  device  was  inoperative.  Thus  a  very  large  number  of 
errors  are  made  in  Situation  4,  and  this  one  type  of  error  is  by 
far  the  most  common  one. 

Summary.  On  many,  but  not  all,  of  the  procedures,  the  model 
group  remembers  instructed  procedures  more  accurately,  uses  more 
efficient  procedures  much  more  often,  and  executes  the  procedures 
faster.  This  is  true  even  after  one  week.  However,  some 
procedures  show  no  effect  of  the  device  model.  Furthermore,  model 
subjects  make  a  different  pattern  of  errors  than  rote  subjects, 
and  make  some  types  of  errors  more  often.  This  rules  out  a 
general  motivational  explanation  for  the  overall  better  retention 
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of  the  model  subjects,  and  it  also  shows  that  the  device  model  can 
actually  impair  some  aspects  of  performance. 

Debriefing  Responses 

At  the  end  of  each  session,  the  subjects  were  asked  to 
describe  the  control  device.  The  protocols  were  analyzed  to 
determine  what  kinds  of  statements  were  made  by  the  subjects  in 
each  group  concerning  the  device  on  a  whole,  and  the  individual 
controls  and  indicators.  Since  these  responses  were  not  collected 
under  carefully  controlled  conditions,  the  quantitative  data  will 
not  be  presented.  Table  10  shows  the  typical  statements,  which 
were  made  by  at  least  15%  of  the  subjects  in  any  one  group  and 
session.  The  rote  group  made  some  consistent  statements  about  the 
device  as  a  whole,  but  most  such  responses  were  idiosyncratic, 
such  as  "you  could  use  it  in  a  plant"  or  "it's  a  security  board". 
In  contrast,  half  to  two-thirds  of  the  model  subjects  made  the 
typical  general  responses  shown.  The  statements  about  individual 
device  features  (controls  and  indicators)  were  typically  made  by 
about  one-quarter  of  the  rote  subjects,  and  one-quarter  to 
one-half  of  the  model  subjects.  This  shows  that  more  of  the  model 
subjects  had  and  retained  a  clearer  understanding  of  the  function 
of  each  control  than  for  the  rote  subjects,  and  also  that  they 
retained  the  device  model  reasonably  well.  But,  many  rote 
subjects  had  acquired  a  basic  understanding  of  the  relations 
between  the  controls.  A  good  example  is  the  power  switch  SP, 
which  was  recognized  as  such  by  the  majority  of  the  rote  subjects. 

Discussion 

The  first  experiment  shows  that  having  a  device  model  does 
improve  performance  on  learning  and  retaining  the  operating 
procedures  for  a  device.  This  result  is  significant  because  it  is 
not  yet  been  clearly  documented  that  such  effects  do  exist. 
However,  the  question  arises  as  to  how  the  device  model  produces 
these  effects.  Clearly  the  extra  knowledge  about  the  device  must 
in  some  way  have  allowed  subjects  to  learn  the  procedures  faster, 
and  then  once  they  were  learned,  to  be  retained  better.  In  some 
ways,  this  result  is  similar  to  that  which  has  been  obtained  in  a 
variety  of  learning  situations  in  which  the  more  "meaningful"  the 
material  is,  the  faster  it  is  learned  and  the  better  it  is 
retained.  However,  to  say  that  the  device  has  become  more 
meaningful,  by  virtue  of  having  explained  how  it  works,  does  not 
actually  answer  the  question  of  how  this  extra  information  makes 
the  procedural  information  easier  to  learn  and  remember.  The  many 
ways  in  which  the  results  were  specific  to  individual  aspects  of 
the  situations  and  the  device  model  suggest  that  the  effects  are 
due  to  very  specific  properties  of  the  model,  and  not  its  global 
effects . 

It  was  claimed  in  the  introduction  that  relevant  how-it-works 
knowledge  is  that  which  can  support  inferences  about  the  operating 
procedures  for  the  device.  This  hypothesis  also  provides  a  direct 
explanation  for  why  it  would  be  easier  to  learn  to  operate  the 
device  and  to  remember  the  procedures  once  they  are  learned. 
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Table  10 


Typical  Statements  about  Device 
Controls  Made  by  Each  Group 


Rote  Group 

Device  as  a  Whole: 

Device  is  controller,  or  tester 
Computer  controls  device 
Don't  know  what  device  is  for 

Device  Features: 

SP  is  power  switch 
SPI  is  power  indicator 

Indicators  show  malfunctions,  everything  works 

EBI,  MAI  show  malfunctions 

PFI  shows  success 

PFI  activated  by  buttons 

ESS-N  is  neutral  or  "off" 

ESS  selects  different  paths  or  circuits 
ESS-MA  works  with  FM ,  and  ESS-SA  with  FS 

Model  Group 

Device  as  a  Whole: 

Device  is  for  firing  phasers 
Description  of  power  flow  through  device 

Device  Features: 

SP  is  power  switch 

SPI  is  power  indicator 

EBI  is  energy  booster  indicator 

MAI  show  main  accumulator  working 

No  indicator  for  secondary  accumulator 

PFI  is  phaser  firing  indicator 

PFI  shows  success 

ESS-N  is  neutral  or  "off" 

ESS  selects  an  accumulator,  energy  source 
ESS-MA  works  with  FM ,  and  ESS-SA  with  FS 
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Namely,  it  is  an  old  and  established  principle  of  learning  that  if 
the  learner  can  reconstruct  the  to-be-remembered  information  in 
more  then  one  way,  then  learning  and  memory  will  be  facilitated. 
Perhaps  the  device  model  in  Experiment  1  simply  allowed  subjects 
to  infer  the  procedures  for  operating  the  device  much  more  readily 
than  the  rote  subjects  could,  and  also  allowed  the  model  group 
subjects  to  reconstruct  by  means  of  inference  the  operating 
procedures  even  if  specific  details  of  the  direct  memory  of  them 
had  been  forgotten.  This  provides  a  specific  explanation  for  how 
making  a  device  "meaningful"  allows  it  to  be  learned  ana 
remembered  better. 

An  obvious  implication  of  this  hypothesis  is  that  if  subjects 
are  not  asked  to  learn  a  set  of  procedures,  but  rather  to  infer 
them,  the  model  group  subjects  should  have  a  decided  advantage. 
Experiment  2  tested  this  implication  by  using  two  groups  ns 
before,  in  which  one  learned  the  device  model  and  the  other  die 
not,  and  they  were  asked  to  iifer  the  procedures  while  "thinking 
out  loud".  It  was  expected  that  the  subjects  with  the  device- 
model  would  be  able  to  infer  the  procedures  quite  readily,  and 
that  the  think-aloud  protocol  data  would  show  that  they  were 
basing  their  inferences  on  the  device  model.  In  contrast,  the 
rote  group  would  be  forced  to  rely  either  on  very  general  aspects 
of  how  one  operates  equipment,  such  as  the  fact  that  it  has  to  be 
turned  on,  or  would  follow  some  kind  of  trial  and  error  procedure. 

EXPERIMENT  2 
Method 

Subjects .  Subjects  were  students  of  both  sexes  at  the 
University  of  Arizona,  recruited  through  campus  advertisements. 
Subjects  were  paid  $5.00  for  participating  in  the  experiment.  Of 
the  11  subjects  who  participated,  the  data  of  one  subject  was 
discarded  because  the  verbal  report  was  inaudible  on  the 
recording.  Of  the  subjects  whose  data  was  used,  there  were  5 
subjects  in  each  condition,  making  a  total  of  10.  One  subject  in 
each  condition  was  female.  Subjects  were  run  individually,  being 
assigned  to  their  conditions  at  random. 

Instructions  and_  procedur e .  The  equipment  and  general 
procedure  were  the  same  as  for  Experiment  *10.  Subjects  first  read 
the  instructions  for  the  experiment  from  the  video  terminal.  For 
the  rote  subjects,  the  instructions  explained  the  general  purpose 
of  the  experiment,  allowed  the  subjects  to  become  familiar  with 
the  layout  of  the  device,  and  then  instructed  subjects  on  the 
experimental  procedure.  The  model  subjects  were  given  the  same 
instructions,  except  that  before  the  instructions  on  the 
experimental  procedure,  they  read  the  device  model  materials  and 
did  the  quiz,  as  in  Experiment  1. 

Except  for  the  experimental  procedure,  the  initial 
instructions  were  very  similar  to  Experiment  1.  Small 
modifications  in  the  instructions  were  made  to  ensure  that 
subjects  were  given  no  clues  about  the  device  or  procedures.  The 
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two  main  changes  were:  first,  in  the  familiarization  material  in 
Experiment  1,  the  SP  switch  was  described  as  being  "on"  when  it 
was  in  the  up  position;  this  was  changed  so  that  no  mention  of 
the  SP  switch  being  "on"  or  "off"  was  made.  Second,  the  story  ana 
quiz  that  were  given  to  subjects  in  the  model  condition  were 
carefully  examined  and  any  suggestions  of  a  particular  order  in 
which  switches  should  be  thrown  or  buttons  pressed  were 
eliminated.  This  resulted  in  no  changes  in  the  wording  of  the 
quiz,  and  only  a  few  minor  changes  in  the  wording  of  the  story. 
Also,  since  subjects  had  to  infer  the  operating  procedures,  it  was 
important  for  them  to  have  a  consistent  starting  state  for  the 
device.  An  addition  was  made  to  the  instructions  following  the 
fam il iar i zation  with  the  device  layout.  The  device  was  described 
as  being  in  its  initial  or  starting  state  when  the  SP  switch  was 
down  and  the  ES  selector  was  at  N,  and  subjects  were  asked  to 
memorize  this. 

The  instructions  told  rote  subjects  that  the  goal  of 
operating  the  device  was  to  make  the  PF  indicator  flash,  and  that 
they  themselves  would  discover  the  control  settings  that  make  this 
happen.  They  were  told  that  we  expected  them  to  use  trial  and 
error  to  find  the  settings  that  would  work.  Because  the  purpose 
of  the  experiment  was  to  find  out  what  reasoning  they  would  use  to 
make  the  device  work,  they  should  "think  aloud"  while  they  were 
working,  and  we  were  particularly  interested  in  their  guesses, 
hypotheses,  and  the  knowledge  they  were  using  to  find  the  control 
settings  that  would  work.  After  they  had  found  settings  that 
would  make  the  PF  indicator  flash,  they  would  then  develop  a 
procedure  to  make  the  PF  indicator  flash  in  as  few  steps  as 
possible.  They  would  then  be  given  an  opportunity  to  practice 
their  procedure.  There  was  also  a  second  way  to  make  the  PF 
indicator  flash  which  they  would  discover  and  practice  after  they 
had  found  their  first  way. 

Instructions  to  the  model  subjects  were  similar  to  those 
given  to  the  rote  subjects,  except  that  they  were  told  that  the 
goal  of  operating  the  device  was  to  make  the  phasers  fire,  rather 
than  make  the  PF  indicator  flash.  The  model  subjects  had  the 
device  diagram  displayed  above  the  device  throughout  the  whole 
exper iment . 

After  subjects  had  read  the  instructions ,  the  experiment  was 
begun,  with  all  statements  and  activities  of  the  subject  recorded 
on  videotape;  the  lab  computer  recorded  each  change  of  the 
controls  on  the  device.  The  experimenter  prompted  the  subjects 
with  questions  if  necessary  to  encourage  "thinking  out  loud".  The 
subjects  inferred  the  procedures  in  two  phases;  the  first 
consisted  of  inferring  the  two  normal  situation  procedures 
(Procedures  1  and  2  in  Tables  1  and  2);  the  second  phase 
consisted  of  inferring  the  malfunction  situation  procedures. 

To  begin  the  first  phase,  the  subject  was  instructed  to  make 
the  PF  indicator  flash.  After  the  subject  had  succeeded,  the 
procedure  that  he  or  she  had  used  was  given  a  name,  depending  on 
which  control  settings  had  been  used.  For  example,  a  procedure 


Page  IP 


using  the  SA  setting  of  the  ESS  selector  (Procedure  ?  in  Table  ?) 
was  given  the  name  SA  proced ur e .  The  subject  then  worked  out  a 
second  procedure  for  making  the  PF  indicator  flash  and  practiced 
it.  This  was  then  given  the  corresponding  name,  for  example,  the 
MA  procedure .  These  names  for  the  "normal”  procedures  were  used 
in  the  rest  of  the  session  to  command  the  subject  to  attempt  a 
particular  procedure. 

After  they  had  established  the  two  procedures  for  making  the 
device  work,  subjects  began  the  second  phase.  They  were  given 
instructions  that  the  device  would  sometimes  break  down. 
Sometimes  it  would  be  possible  to  change  the  control  settings  and 
still  make  the  PF  indicator  flash,  but,  at  other  times,  the 
malfunction  would  mean  that  the  PF  indicator  could  not  be  made  to 
flash.  Therefore,  if  the  device  did  break  down,  they  should  make 
the  PF  indicator  flash,  if  they  could,  and  if  it  could  not  be  made 
to  flash,  they  should  tell  the  experimenter.  Sometimes  the  device 
would  work  perfectly,  and  at  other  times  there  would  be  a 
malfunction.  The  indicator  lights  might  be  helpful  to  them  in 
deciding  what  type  of  malfunction  had  occurred,  but  this  would  not 
always  be  true. 

In  the  second  phase  of  the  experiment,  subjects  did  two  or 
more  runs ;  in  each  run,  there  were  16  trials,  in  which  they  were 
asked  by  the  experimenter  (cued  by  the  lab  computer)  to  operate 
the  device  using  either  the  MA  or  SA  procedures.  The  device 
worked  normally  half  the  time,  and  each  of  the  eight  malfunction 
situations  appeared  once.  The  malfunction  situations  were  the 
same  as  for  Experiment  1,  and  are  shown  in  Tables  1  and  2.  ^he 
order  of  malfunctions  was  the  same  for  every  subject.  In  the 
first  run,  the  order  in  which  malfunctions  appeared  was  designed 
to  introduce  the  malfunctions  in  rough  order  of  increasing 
difficulty.  The  first  situation  that  the  subject  saw  was  one 
where  the  device  worked  normally.  The  first  malfunction  was 
chosen  to  be  one  where  the  EB  and  MA  indicator  lights  were  both 
off  and  the  device  could  not  work.  Situations  where  the  device 
could  be  made  to  function  by  using  the  control  settings  for  the 
alternate  procedure  from  the  one  that  they  were  asked  to  do, 
Situations  5  and  9,  appeared  before  the  correspond ing  situations 
where  the  alternate  method  would  not  work,  Situations  6  and  10. 
Normal  and  malfunction  situations  alternated,  but  the  order  of 
which  procedure  (MA  or  SA)  was  commanded  was  apparently  random. 

Subjects  did  a  minimum  of  two  runs;  all  the  rote  subjects 
did  three  runs,  as  did  one  of  the  model  subjects.  The  other  four 
model  subjects  did  two  runs  only.  For  the  second  and  third  runs, 
the  order  in  which  the  malfunctions  appeared  was  a  random  order, 
but  was  the  same  for  every  subject.  After  one  or  two  runs  through 
the  malfunctions,  subjeccs  were  asked  to  be  as  efficient  as 
possible  in  their  decisions  as  to  whether  they  could  make  the 
device  function  or  not.  For  the  model  subjects,  this  took  the 
form  of  asking  the  subjects  to  pretend  that  the  Klingons  were 
attacking  the  Enterprise ,  and  so  the  phasers  should  be  fired  if  it 
was  at  all  possible.  17  there  was  a  malfunction,  or  if  the 
phasers  could  not  be  made  to  fire,  then  they  should  make  a  report 
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to  engineering  on  what  the  problem  seemed  to  be  as  soon  as 
possible.  Rote  subjects  were  simply  asked  to  predict  what  they 
thought  was  going  to  happen,  and  to  tell  the  experimenter  as  soon 

as  possible  if  they  thought  there  was  a  malfunction.  During  these 
"efficiency"  runs,  subjects  were  frequently  asked  by  the 
experimenter  where  they  thought  the  problem  lay  in  the  device. 

The  number  of  runs  required  of  each  subject  was  based  on  a 
subjective  judgment  by  the  experimenter  as  to  whether  the 
subjects'  performance  had  become  stable  in  all  the  situations. 
The  criteria  used  to  make  this  judgement  were  as  follows:  (a)  The 
subject  was  doing  the  procedures  correctly;  if  the  PF  indicator 
could  be  made  to  flash  then  the  subject  was  able  to  do  it,  and  if 
the  PF  indicator  could  not  be  made  to  flash,  the  subject  could 
recognize  this  and  tell  the  experimenter.  (b)  There  was  little 
hesitation  when  the  subject  was  faced  with  a  malfunction 
situation.  Such  hesitation  was  exhibited  if  the  subject  continued 
to  experiment  with  various  settings,  or  made  explicit  verbal 
statements.  For  3  of  the  5  rote  group  subjects,  these  criteria 
were  judged  not  to  be  met  before  the  subject’s  time  ran  out. 

Results 


Number  of  Actions  Tried 

The  computer  recorded  every  change  in  a  switch  position  or 
button  press.  Each  change  defined  an  action  made  by  a  subject. 
Note  that  because  the  ESS  switch  requires  going  through  the  SA 
position  going  to  and  from  the  MA  position,  the  normal  MA 
procedure  requires  2  more  actions,  a  total  of  7,  than  the  SA 
procedure,  which  requires  5.  The  number  of  actions  attempted 
while  inferring  the  MA  and  SA  procedures  is  shown  in  Table  11, 
which  shows  the  mean  number  of  actions  made  on  each  attempt  for 
the  two  procedures.  Both  groups  ended  up  with  the  same  optimal 
procedures,  but  the  rote  group  tried  a  very  large  number  of 
actions  in  their  first  and  second  attempts.  In  contrast,  the 
model  subjects  deduced  the  correct  procedures  almost  exactly 
correctly  on  their  first  attempt.  This  difference  in  number  of 
steps  on  the  first  attempt  between  the  two  groups  is  significant 
(_t(8)  =  3<75,  ja<  .01).  Notice  that  most  of  the  rote  group  discovered 
the  SA  procedure  first,  since  it  is  the  first  setting  on  the  ES 
selector  switch,  while  the  model  group  discovered  the  MA  procedure 
first,  since  it  corresponds  to  the  "Main"  accumulator. 

The  number  of  actions  tried  while  inferring  the  procedure  for 
each  malfunction  situation  on  each  run  is  shown  in  Table  12. 
Because  the  number  of  runs  through  the  malfunction  situations  was 
different  for  the  two  instruction  conditions,  two  separate 
three-way  analyses  of  variance  were  performed;  the  first  analysis 
used  the  number  of  actions  on  the  first  and  second  runs,  and  the 
second  analysis  used  the  number  of  actions  on  the  first  and  final 
(second  or  third)  runs. 
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Table  11 


Mean  Number  of  Actions  Tried  While 
Inferring  Normal  Procedures 


Attempt 


Group 

Procedure  Inferred 

1 

2 

3 

Rote 

MA 

first 

14.0(1) 

7.01(1) 

SA 

second 

10.0(1) 

5.0 

(1  ) 

SA 

first 

26.0(4) 

5.3 

(4) 

5.0(1  ) 

MA 

second 

9.3(4) 

7.0 

(2) 

Model 

MA 

first 

7.4(5) 

7.0 

(2) 

SA 

second 

5.0(5) 

5.0 

(1  ) 

Note:  The 

numbers  in  parentheses 

are  the 

number  of 

subjects  contributing  to  the  mean. 


Table  12 


Mean  Number  of 
Run  for  Each 

Actions  Tried  in  each 

Group  and  Situation 

RUN 

Situation 

1 

2 

3 

ROTE  GROUP 

3 

46.6 

9.2 

7.2 

4 

21  .8 

14.4 

12.0 

5 

16.8 

9.2 

8.0 

6 

24.2 

12.0 

8.2 

7 

25.0 

12.4 

6.2 

8 

18.4 

10.0 

7.2 

9 

18.0 

8.4 

8.4 

10 

14.0 

10.2 

8.0 

MEAN 

23.1 

10.7 

8.2 

MODEL  GROUP 

3 

9.8 

7.2 

4 

9.4 

11.6 

5 

9.4 

6.8 

6 

9.0 

8.0 

7 

7.2 

5.6 

8 

8.8 

8.0 

9 

8.2 

8.4 

10 

11.6 

11.6 

MEAN 

9.2 

8.4 
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The  three-way  ANOVA  on  the  first  and  second  run  data  shows 
that  there  was  a  main  effect  of  instructional  condition 
( F( 1 , 8 ) = 1 2 . 35 ,  £<.01).  The  mean  number  of  actions  for  model 
subjects  was  8.8,  and  for  rote  subjects  was  16.9;  thus,  the  rote 
subjects  took  92%  more  actions  than  the  model  subjects.  There  was 
a  main  effect  of  run  ( F( 1 , 8 ) =24 . 91 ,  p<.01),  with  the  mean  number 
of  actions  on  the  first  run  being  16.1,  and  on  the  second  run, 
9.6.  There  was  also  a  main  effect  of  situation  ( F(7 , 56 ) =2. 38 , 
p<.05).  All  three  of  the  two-way  interactions  were  significant. 
As  shown  in  Table  12  the  decline  in  number  of  actions  from  Run  1 
to  Run  2  is  small  for  the  model  group,  but  is  substantial  for  the 
rote  group  ( F( 1 , 8) = 1 9 . 38 ,  £<.01).  For  the  model  group  there  was 
relatively  little  difference  between  the  number  of  actions  done  in 
inferring  the  procedure  for  different  situations,  but  for  the  rote 
group,  the  difference  was  substantial.  This  group  by  situation 
interaction  was  significant.  ( F(7 , 56 )  =  3  .  1 2  ,  p<.01).  The  run  by 
situation  interaction  was  also  significant  ( F( 7 , 56 ) =2 . 68 ,  p<.05), 
but  means  only  that  some  situations  did  not  require  as  many 


actions  on  the 
others . 

first  run  relative  to 

the 

second 

one  , 

as  did 

The  ANOVA 

on  the  first  and  final 

runs 

showed 

very 

similar 

effects  to  those  seen  for  the  first  and  second  runs.  There  were 
main  effects  of  instructional  condition  ( F( 1 , 8 )  =  9 •  1 6 , p<  .  05 )  ,  run 
( F(  1 , 8)  =  35 . 98 ,  £<.01)  but  not  of  situation  ( F( 7 , 56 )  =  2 .  1  ,  p=.059  ). 
The  three  two-way  interactions  are  similar  to  those  above,  between 
condition  and  run  ( F( 1 , 8)=29. 23 ,  £<.01),  condition  and  situation 
(F(7,56)=2.56,  p<.057,  and  run  and  situation  ( F( 7 , 56 ) =3 • 1 9 , 
£<  .01).  There  is  also  the  three-way  interaction  ( F( 7 , 56 )  =  2 . 33 , 
£<.05).  Generally  speaking,  the  overall  picture  is  the  same  as 
with  the  first  and  second  run  data,  but  since  the  rote  group  third 
run  is  nearly  identical  to  the  model  group  second  run,  the 
interaction  effects  are  perhaps  more  pronounced.  This  final 
similarity  shows  that  the  group  differences  is  not  associated  with 
the  efficiency  of  the  final  procedures,  but  rather  with  the 
efficiency  of  inferring  the  procedures. 

Examination  of  the  effects  of  the  order  in  which  subjects  saw 
the  situations  shows  that  for  the  model  subjects  there  is 
virtually  no  effect  of  order;  these  subjects  try  almost  the  same 
number  of  actions  on  the  first  malfunction  situation  on  their 
first  run  as  they  do  for  the  last  malfunction  situation  on  their 
final  run.  However  for  rote  subjects,  the  picture  is  quite 
different.  On  the  very  first  malfunction  situation  that  they  see, 
they  try  far  more  actions  than  for  any  other  situation.  They  also 
try  more  actions  generally  on  all  situations  for  their  first  run 
than  on  their  second  or  final  runs. 
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Pushbutton  Actions 

One  type  of  action  is  very  diagnostic  of  the  approach  used  by 
subjects  while  inferring  the  procedures;  this  is  pushing  the 
buttons  FM  and  FS.  The  button  pushes  made  by  the  subject,  for  each 
situation  were  classified  by  type.  Table  13  shows  the  proportion 
of  responses  of  each  type  in  each  condition.  Tn  CORRECT 
responses,  the  SP  switch  was  on  and  the  button  pushed  was 
appropriate  for  the  setting  of  the  ES  selector.  In  CROSS 
responses,  the  SP  switch  was  on  and  the  ES  selector  was  set  to  the 
opposite  accumulator  for  the  button  pushed,  for  example,  pressing 
the  FM  button  with  the  ES  selector  set  to  SA.  In  NEUTRAL 
responses,  the  SP  switch  was  on  and  the  ES  selector  was  set  to 
neutral  when  either  one  of  the  buttons  was  pressed.  In  OFF 
responses,  the  SP  switch  was  off  when  the  button  was  pushed  with 
the  ES  selector  in  any  position.  In  BOTH  responses,  both  buttons 
were  pressed  simultaneously  and  the  other  controls  were  in  any 
position.  Finally,  HOLD  responses  involved  one  or  both  of  the 
buttons  being  held  down  while  other  switches  were  manipulated. 
Notice  that  in  terms  of  the  device  model,  all  of  these  response 
types,  except  for  CORRECT,  are  "nonsense"  responses. 

Log-linear  analysis  of  the  button-push  type  on  the  first  two 
runs  for  each  instructional  condition  shows  that  the  model  of  best 
fit  includes  all  main  effects  and  two-way  interactions  (all 
£s<.05),  but  no  three-way  interaction.  The  two-way  tables, 
expressed  with  proportions,  are  shown  in  Table  14.  The  table  for 
the  condition  by  button- push  type  interaction  shows  that  model 
subjects  are  correct  more  often  than  rote  subjects,  and  are 
incorrect  less  often,  particularly  in  CROSS  button  pushes.  The 
run  by  type  interaction  means  that  subjects  more  often  use  CORRECT 
button  pushes  on  their  second  run  then  on  their  first,  while  the 
incorrect  button  pushes  decline  in  frequency  on  the  second  run. 
This  is  particularly  true  for  HOLD,  BOTH,  OFF  and  NEUTRAL  while 
CROSS  is  reduced  only  slightly.  The  run  by  condition  interaction 
shows  that  model  subjects  do  not  change  in  number  of  button  pushes 
between  first  and  second  runs  very  much  (35/E  reduction)  compared 
to  rote  subjects  (150%  reduction).  But  when  the  final  runs  only 
are  compared  (run  2  for  model  subjects,  run  3  for  rote  subjects), 
there  is  no  significant  difference  in  the  distribution  of 
responses  for  model  and  rote  subjects  (X~2  =4.638,  p>.2,  5df) .  A 
log-linear  analysis  of  the  push-type  data  for  the  first  and  final 
runs  shows  a  similar  pattern  to  that  described  above.  However, 
the  effects  are  weaker,  because  the  final  runs  for  both 
instruction  conditions  are  similar,  and  the  first  and  final  runs 
for  model  subjects  are  also  similar. 


Table  13 


Proportions  of  Button-Push  Type 
for  each  Run  for  each  Group 


RUN 


1 

2 

3 

Total 

ROTE  GROUP 

Correct 

.43 

.65 

.86 

.55 

Cross 

.21 

.20 

.08 

.  19 

Neutral 

.18 

.  1 1 

.05 

.  14 

Off 

.13 

.04 

.01 

.09 

Both 

.05 

0 

0 

.03 

Hold 

.005 

0 

0 

.003 

N 

377 

150 

87 

614 

MODEL  GROUP 

Correct 

.85 

•  94 

.89 

Cross 

.02 

.01 

.02 

Neutral 

.04 

.05 

.05 

Off 

.07 

0 

.04 

Both 

.02 

0 

.  01 

Hold 

0 

0 

N 

115 

85 

200 

Table  14 


Interactions  of  Group,  Run, 
and  Button-Push  Type 


Condition 


Button-push  type 

Rote  (N=527) 

Model  ( N=200 ) 

Correct 

.50 

.89 

Cross 

.21 

.02 

Neutral 

.16 

.05 

Off 

.10 

.04 

Both 

.04 

.003 

Hold 

.003 

0 

Run 

Button-push  type 

One  (N=492) 

Two  (N=235) 

Correct 

.53 

.76 

Cross 

.16 

.  13 

Neutral 

.14 

.09 

Off 

.11 

.03 

Both 

.04 

0 

Hold 

.004 

0 

Condition 

Run 

Rote  ( N=527 ) 

Model  ( N=200) 

One 

.71 

.58 

Two 

.28 

.43 
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Protocol  Analysis 


The  subject's  think-aloud  protocols  were  transcribed,  and 
information  included  on  which  situation  the  subject  was  working 
on,  along  with  the  specific  control  actions  the  subject  performed 
while  talking  aloud.  The  first  step  in  the  analysis  was  to 
eliminate  redundant  or  irrelevant  content  in  the  protocols.  This 
was  done  by  summarizing  for  each  subject  the  statements  that  the 
subject  made  about  the  device  or  about  the  experimental  situation. 
These  statements  were  then  classified  into  a  set  of  major 
categories,  and  then  the  statements  within  each  category  were 
classified  by  content.  The  major  categories  used  were  (a)  reason 
statements  given  by  the  subject  for  performing  a  specific  control 
action,  (b)  hypothesi s  statements  about  the  device  as  a  whole, 
individual  controls  or  indicators,  or  about  particular  malfunction 
situations  and  their  causes,  (c)  feature  statements  by  the  model 
subjects  made  about  the  function  of  individual  controls, 
indicators,  and  aspects  of  the  device,  (d)  meta-statements  about 
the  device  or  experiment  as  a  whole,  and  (e)  af feet i ve  statements 
about  what  the  subject  was  feeling.  Once  the  statements  were 
classified  by  content,  a  tally  was  made  of  which  subjects 
generated  statements  in  each  content  category.  Since  some 
subjects  are  considerably  more  talkative  than  others,  what  was 
tallied  was  which  subjects  made  at  least  one  statement  in  each 
given  content  category.  Thus  if  a  subject  made  several  statements 
with  essentially  the  same  content,  that  subject  was  only  counted 
as  having  made  one  such  statement,  giving  a  total  of  297  subject 
statements.  For  purposes  of  presentation  here,  the  different 
types  of  statements  were  further  reduced  into  a  small  number  of 
categories.  Tables  15,  16,  and  17  show  each  statement  type  along 
with  one  or  more  examples  of  statements  included  in  that  type,  and 
the  number  of  subjects  making  these  statements,  along  with  the 
mean  number  of  times  the  statements  were  made  by  these  subjects. 
By  considering  both  the  number  of  subjects  making  statements, 
along  with  the  average  number  of  statements  made  by  these 
subjects,  a  reasonable  picture  can  be  gained  of  both  the  amount 
and  consistency  of  statements  made  by  the  two  groups. 

Table  15  shows  the  results  for  the  reasons  given  for 
performing  actions  while  inferring  the  procedures.  The  table  is 
in  two  parts  corresponding  to  the  first  phase  of  the  procedure 
inference  process,  in  which  subjects  inferred  the  MA  and  SA 
procedures,  and  the  second  phase  in  which  they  inferred  the 
procedures  for  the  various  malfunction  situations.  As  shown  in 
the  table,  the  rote  group  explained  their  actions  primarily  in 
terms  of  general  features  of  the  device,  and  some  form  of 
systematic  search  strategy  such  as  working  from  left  to  right  or 
trying  all  combinations.  In  contrast,  the  model  group  gave  a 
large  number  of  reasons  based  on  the  device  model.  A  similar 
pattern  appears  with  the  malfunction  procedures.  The  rote  group 
mak«s  many  statements  based  on  either  behavioral  features  of  the 
device  or  based  on  their  search  strategy.  In  contrast,  the  model 
group  gave  many  reasons  for  operating  controls  based  on 
malfunctions  in  the  internal  components  on  the  device.  Other 
features  of  these  reasons  will  be  returned  to  below. 


Table  15 


Mean  Number  of  Reasons  Given  for 
Performing  Actions  While  Inferring  Procedures 


Group 

Type  of  Reason  and  Examples 

Rote 

Model 

MA  and  SA  Procedures 

General  Device  Feature: 

"SP  seems  like  an  ON  switch” 

1.5 

(2) 

0 

Search  Strategy: 

"That's  working  left  to  right" 

"MA  is  only  thing  left" 

3.0 

(5) 

0 

Model -based  : 

"That  applies  power  from  ship  to 
phasers" 

0 

4 . 0 

Mai  function  l’r  etur.-s 

Sear  ■  hi  St  • 

"  Tr  :  a  :  a-"  :  '  r" 

1.75 

(«  ) 

0 

Model  - 1  a  r-  :  . 

"Mair.  v  ^jr  .  •  '  .  «  not  woking" 

0 

2.8 

(5) 

De  v  i  he  •  •  .  .  »  : 

"SA  d  .  m  '  •  -  r-  , 

"  E  P  ’  :  s  >  r. " 

?.  ? 

(5) 

1  •  3 

(3) 

Ind  i  :  at  ->r  -  >  .  1  e  : 

n 

l 

o 

Ou  1 

1 

1 

(3) 

Note:  Number  1  r,  parentheses  is  the  number  of  subjects 

contributing  to  the  mean. 


Table  16  shows  the  statements  that  contained  explanations  an  i 
hypotheses  about  properties  of  the  device  or  causes  of  the 
malfunctions.  These  statements  classify  into  rules,  model-based 
explanations,  features  of  the  device,  and  hypotheses  about  causes 
of  malfunctions.  Examination  of  the  table  shows  that  although  the 
rote  group  makes  many  more  statements  in  the  form  of  rules  and  in 
terms  of  specific  features  of  the  device,  the  model  group  provides 
a  large  number  of  explanations  and  hypotheses  based  on  the  device 
model  . 

Finally,  Table  17  shows  the  meta-statements  and  affective 
statements.  It  is  clear  that  certain  kinds  of  statements  are  much 
more  likely  to  be  made  by  the  rote  group  than  the  model  group. 
The  rote  group  made  many  more  statements  about  the  device  behaving 
in  an  unreliable  or  arbitrary  fashion,  or  about  the  experimental 
situation  being  deliberately  tricky,  or  that  the  subject  was 
confused  or  frustrated.  In  contrast,  the  model  group  often  made 
statements  that  corresponded  to  the  Star  Trek  fantasy  underlying 
the  device  model  . 

Discussion 

Having  the  device  model  available  while  inferring  the 
procedures  for  operating  the  device  produced  very  powerful 
differences  in  both  the  performance  of  the  subjects  and  in  the 
protocol  data.  The  group  with  the  device  model  available  inferred 
the  procedures  for  operating  the  device  in  almost  the  minimum 
amount  of  time  possible,  using  almost  the  fewest  trial  actions 
possible  to  do  so.  They  engaged  in  very  few  "nonsense"  trial 
actions,  thus  making  only  attempts  that  were  consistent  with  the 
device  model.  One  of  the  few  cases  where  this  was  not  so  is  one- 
subject  who  took  nine  trial  actions  to  infer  the  first  procedure, 
as  opposed  to  the  minimum  of  seven,  which  was  the  case  the  rest  of 
the  model  group.  This  subject  first  tried  to  fire  the  phasers 
without  first  powering  up  the  system,  saying  as  he  did  so  that  he 
wanted  to  see  if  the  accumulators  still  had  a  stored  charge  that 
could  be  used  to  fire  the  phaser.  Thus,  even  when  a  model  subject 
did  not  perform  optimally,  he  did  so  on  the  basis  of  reasonable- 
inferences  from  the  device  model.  The  fact  that  the  device  model 
was  used  as  a  basis  for  inferring  the  procedures  is  very  clear 
from  the  reasons  pbven  for  the  specific  trial  actions  that  the 
model  group  usee  These  explanations  are  almost  always  in  terms 
of  properties  of  the  model,  in  sharp  contrast,  to  the  rote  group 
whose  activities  are  based  mainly  on  some  type  of  trial  and  error 
search  strategy. 

An  additional  feature  of  having  the  device  model  seems  to  be 
that  it  allows  much  more  consistent  and  coherent  explanations  for 
the  properties  of  the  device  and  its  behavior.  That  is,  the  model 
group  could  give  fairly  consistent  explanations  for  the  device 
behavior  in  terms  of  the  model,  whereas  the  :  ■.  te  group  had  to  use 
many  more  specific  rules  for  the  device  behavio-  and  had  to  refer 
more  often  to  specific  features  of  the  device. 


Table  16 


Mean  Number  of  Explanations  and  Hypotheses 
Stated  About  Device  Properties  and  Causes 
of  Malfunctions 


Hypothesis  Type  and  Examples 

Rote 

Group 

Model 

Rules 

Correct: 

"If  SP  down,  nothing  will  work" 

"If  EBI  off,  then  device  won't  work" 

5.0 

(5) 

1  .  4 

(5) 

Incorrect : 

"If  only  EBI  on,  asked  for  procedure 
won't  work,  but  other  may" 

"Second  in  a  row  won't  work" 

1.75 

(4) 

0 

Model-based  Explanations 

Correct : 

"EBI  shows  if  energy  booster  working" 
"Energy  from  accumulators  is 
transferred  to  phasers" 

0 

1 

1 . 0 

(5) 

Incorrect : 

"EBI  off  means  energy  booster  not 
getting  power  from  ship" 

"ESS  to  MA  turns  on  main  accumulator" 

0 

2.0 

(4) 

Device  Features 

Correct : 

"SPI  is  power  indicator" 

"MAI  is  related  to  MA  on  ESS" 

CO 

(5) 

1  .  0 

(2) 

Incorrect: 

"Indicators  can  malfunction" 

1  .  5 

(2) 

1  .  3 

( 3  i 

Malfunction  Hypotheses 

Device  Features: 

"ESS  is  switched" 

"FS  button  not  working" 

4.7 

(3) 

0 

Mod  el -ba  sed  : 

"Both  accumul ator s  are  malfunctioning" 
"Main  accumulator  not  getting  power 
from  booster" 

0 

6.4 

(5) 

Note :  Number  in  parentheses  is  the  number  of  subjects 

contributing  to  the  mean. 
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Table  17 

Mean  Number  of  Meta  Statements 
and  Affective  Statements  Made 
During  Experiment 


Group 


Statement  Type  and  Examples  Rote  Model 


Dev  ice  Unr el iability :  2.8(5)  1.5(2) 

"It  seems  totally  random" 

"No  way  to  tell  if  it’ll  work" 

Ex  per imenter ' s  Tr ic  ks :  2.3(3)  0 

"You  might  be  tricking  me” 

"You  guys  are  changing  the  rules" 

Star  Trek"  0  1.0(4) 

"Get  Spock,  he  can  figure  it  out" 

Subject ' s  Conf  usion  ,  Fr  us tr  at i on :  3-0(4)  0 

"I  can’t  understand  it,  I’m  thoroughly 
confused " 

"At  this  stage,  I’d  throw  the  machine 
out  of  the  window" 

Note :  Number  in  parentheses  is  the  number  of  subjects 

contributing  to  the  mean. 


Finally,  the  way  in  which  the  device  model  affected  the 
subject’s  overall  view  of  the  situation  and  its  affective  content 
is  striking.  The  group  without  the  model  entertained  statistical 
hypotheses,  such  as  that  every  other  trial  would  work,  and  also 
that  the  device  was  unreliable,  or  that  the  experimenter  was 
tricking  them,  and  expressed  much  more  confusion  and  frustration 
with  the  device.  Perhaps  the  device  model  makes  possible  direct 
and  easy  inferences  about  operating  the  device,  and  so  kept  the 
very  same  behavior  of  the  device  from  looking  arbitrary  or  subtle. 

There  are  some  small  features  of  these  protocol  analyses  that 
can  be  pointed  out.  For  the  rote  group,  there  is  some  indication 
that  they  were  using  a  stereotyped  layout  schema  for  the  control 
panel.  Several  of  the  rote  group  worked  systematically  from  left 
to  right.  For  example,  the  SP  switch  seemed  to  be  readily 
recognized  as  a  power  switch.  That  position  for  the  SP  switch  as 
being  the  left-most  position  may  have  contributed  to  its 
perception  as  being  the  power  switch.  One  or  two  subjects  made 
statements  to  the  effect  that  all  of  the  switches  and  controls  on 
the  device  must  be  there  for  a  reason,  and  based  their-  search 
strategy  on  this  assumption. 

In  terms  of  explaining  malfunctions  of  the  device,  or 
apparently  inconsistent  behavior  of  the  device,  a  surprising 
result  was  that  subjects  in  both  groups  attributed  defects  either 
to  the  front  panel  parts  of  the  device,  such  as  the  lights  and 
switches,  or  to  the  connections  between  them.  That  is,  relatively 
common  statements  were  that  an  indicator  light  could  be 
malfunctioning,  that  the  wires  in terconnecting  the  lights  and 
switches  could  be  broken,  or  that  the  push  buttons  themselves 
might  not  be  working.  Thus,  for  the  rote  group,  it  is  as  if  there 
were  no  internal  components  of  the  device.  There  is  also  some 
indication  that  the  model  group  tended  to  believe  that  the 
internal  components  of  the  system  did  not  malfunction;  rather, 
the  connections  between  these  components  could  become  defective. 
For  example,  a  very  common  statement  was  one  like  "the  main 
accumulator  is  not  getting  power  from  the  booster"  for  explaining 
why  the  MA  procedure  would  not  work  when  the  MAI  light  was  off. 
To  some  extent,  the  analysis  of  the  device  in  terms  of  unreliable 
connections,  controls,  and  indicators  appears  to  be  isomorphic  to 
the  intended  device  model  analysis  in  terms  of  unreliable  internal 
components.  It  is  interesting  to  speculate  that  in  the  real 
world,  the  most  common  failures  in  electrical  equipment  are  due  to 
causes  such  as  broken  wires,  burnt-out  indicators,  or  defective 
switches.  Perhaps  subjects  were  generalizing  this  experience  to 
this  device. 

Finally,  in  the  model  group,  a  couple  of  subjects  apparently 
had  difficulty  with  the  relationship  of  the  ESS  selector  switch  to 
the  accumulators .  For  example,  a  couple  of  subjects  appeared  to 
believe  that  the  selector  switch  was  between  the  energy  booster 
and  the  accumulators,  since  they  made  statements  such  as  "ESS  to 
MA  turns  on  main  accumulator"  or  "ESS  to  MA  allows  main  to  receive 
power  from  the  booster".  A  similar  confusion  may  underlie  some  of 
the  procedures  that  were  especially  difficult  for  the  model  group 


in  Experiment  1. 


CONCLUSIONS 

These  studies  obtained  definite  and  strong  effects  of  having 
a  device  model.  The  explanation  that  the  task  becomes  more 
"me-.,  ingful"  as  a  result  of  having  a  device  model  can  be  replaced 
with  a  more  specific  explanation,  that  the  device  model  helps 
because  it  makes  possible  specific  inferences  about  what  the 
operating  procedures  must  be.  Thus,  the  basic  principle  advanced 
here  is  that  in  the  context  of  learning  procedures,  relevant 
how-it-works  knowledge  is  the  knowledge  about  the  internal 
workings  of  the  system  that  allows  the  user  to  infer  how  to 
operate  the  device,  given  knowledge  of  the  goals  to  be  achieved 
and  the  basic  operations  that  can  be  performed.  If  this 
definition  of  what  constitutes  a  useful  device  model  is  adopted, 
several  conclusions  follow  directly: 

1.  The  relevant  how-it-works  knowledge  can  be  very 
superficial  and  incomplete.  This  is  because  the  user  does  not 
need  to  have  full  under  stand  ing  of  this  system  in  order  to  be  able 
to  infer  the  procedures  for  operating  it.  Kieras  and  Poison  (Note 
2)  took  this  idea  one  step  further  and  speculated  on  a  set  of 
criteria  that  could  be  used  to  choose  relevant  how-it-works 
knowledge,  based  on  information  about  the  task  goals  and  subgoals 
that  had  to  be  achieved  with  the  device,  and  whether  these  task 
goals  were  already  known  to  the  user  of  the  device,  or  were  in 
fact  specific  to  the  device  being  learned.  The  basic  principle  is 
that  relevant  how-it-works  knowledge  explains  the  system's 
mechanisms  that  are  involved  in  fulfilling  the  user's  goals,  but 
how-it-works  information  that  does  not  explain  how  or  why  a  goal 
must  be  accomplished  is  not  useful.  This  criterion  sets  limits  on 
how  much,  and  how  detailed,  the  information  should  be. 

2.  The  device  model  will  not  always  be  of  value;  it  depends 
on  whether  the  user  in  the  actual  task  situation  both  needs  to 
infer  the  procedures,  and  also  needs  the  information  in  order  to 
be  able  to  infer  the  procedures.  If  the  device  is  very  simple,  or 
the  procedure  is  well  practiced,  there  in  fact  may  be  no  need  for 
such  information. 

One  interesting  side  aspect  of  this  principle  is  that  there 
is  in  fact  a  confounding  between  knowledge  of  device  models  and 
knowledge  of  procedures.  Full  experts  on  using  a  system  usually 
have  both  a  considerable  amount  of  how-it-works  knowledge,  and 
also  a  fully  developed  set  of  procedural  knowledge.  This  tends  to 
suggest  that  having  fully  developed  procedures  depends  on  having  a 
fully  developed  device  model.  But  perhaps  the  fv1 1 y  expert  user 
does  not  make  use  of  a  device  model  except  under  special 
circumstances.  Rather,  procedures  covering  a  wide  range  of  both 
normal  and  abnormal  situations  will  be  well  known  to  the  expert 
user  without  recourse  to  inference  from  the  how-it-works 
knowledge . 
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This  reasoning  suggests  that  the  primary  effect  of  having  a 
device  model  will  be  in  situations  where  previously  learned 
procedural  knowledge  will  not  be  of  direct  value.  Such  situations 
are  those  involved  with  learning  a  novel  device,  troubleshooting, 
operating  the  device  to  achieve  novel  goals,  making  it  operate  in 
spite  of  a  malfunction,  or  devising  more  efficient  procedures  for 
using  the  device.  One  other  use  of  the  ability  to  infer  the 
procedures  would  be  facilitating  performance  after  a  long 
retention  interval.  If  the  details  of  the  procedure  have  been 
lost,  the  device  model  provides  an  alternate  source  of  information 
which  can  be  used  to  reconstruct  the  procedure.  The  point  is  that 
unless  some  of  these  conditions  are  the  case,  there  may  well  be  no 
value  in  having  the  device  model.  This,  for  example,  is  why  the 
user  of  a  telephone  does  not  need  a  device  model  of  the  switching 
network.  The  procedures  for  operating  a  telephone  are  so  limited, 
and  are  so  heavily  driven  by  obvious  aspects  of  the  task,  that  the 
device  model  information  is  simply  of  no  value. 

3.  Learning  and  using  a  device  model  may  have  its  own 
pitfalls.  That  is,  knowledge  of  the  model  may  be  subject  to 
misunderstandings  and  distortions,  like  any  other  knowledge.  Thus 
if  the  user  is  taught  a  device  model,  but  fails  to  learn  it 
correctly,  performance  may  not  be  facilitated  at  all,  or  may 
actually  be  impaired.  This  suggests  that  a  badly  designed  system, 
with  a  device  model  that  is  difficult  to  remember  or  understand, 
may  actually  be  harder  to  learn  if  the  device  model  is  involved. 
Another  possible  pitfall  is  that  a  very  complex  system  may  have  a 
device  model  that  is  so  complex  that  there  is  no  advantage  of 
learning  it.  In  short,  the  costs  of  acquiring  a  correct  and 
usable  device  model  must  be  reckoned  into  any  statements  about  the 
advantages  of  teaching  one  to  the  user. 


Page  27 


Reference  Notes 


1.  Kieras,  D.  E. ,  &  Poison,  P.  G.  An  outline  of  a  theory  of  the 
user  complexity  of  devices  and  systems.  University  of  Arizona 
and  University  of  Colorado,  Working  Paper  No.  1,  May  10,  198?. 


2.  Kieras,  D.  E.,  A  Poison,  P.  G.  An  approach  to  the  formal 
analysis  of  user  complexity.  University  of  Arizona  and 
University  of  Colorado,  Working  Paper  No.  2,  October  1,  1982. 


3.  Bond,  N.  C.  &  Towne,  D.  M.  Troubl eshooti ng  complex  equipment 
in  the  military  services:  Research  and  prospects.  Technical 
Report  No.  92,  Behavioral  Technology  Laboratories,  University 
of  Southern  California,  December  1 97  9  - 


4 .  Kieras,  D.  E.  What  people  know  about  electronic  devices: 
descriptive  study.  Technical  Report  No.  1 

( UARZ/DP/TR-82/ONR-1 2 ) .  University  of  Arizona,  October  1, 
1982. 


References 


Gentner,  D.,  &  Stevens,  A.  L.  (Eds.)  Mental  Models.  Hillsdale, 
N.  J.:  Erlbaum,  1982  (in  press). 


Linton,  M.,  A  Gallo,  P.  The  Practical  Statistician.  Monterey, 
California:  Brooks/Cole,  1975. 


Norman,  D.A.  Some  Observations  on  Mental  Models.  In  D.  Gentner 
and  A.  Stevens  (Eds.),  Mental  Models.  Hillsdale,  New  Jersey: 
Erlbaum,  1982. 


r 


J\) 


APPENDIX  A 


Device  Model  Instructions 

To  help  make  the  device  meaningful  to  you,  we  have  based  it  or; 
Gene  Roddenberry ' s  "Star  Trek".  I  will  explain  how  the  device 
works  in  terms  of  this  fantasy.  This  device  is  a  phaser-bank 
control  from  the  Starship  "Enterprise",  and  you  will  learn  several 
procedures  for  firing  the  phasers.  When  the  phasers  are  to  be 
fired,  the  bridge  will  direct  you  to  fire  using  a  specified 
procedure.  You  will  then  carry  out  the  specified  procedure,  and 
when  you  have  completed  it,  you  will  signal  the  bridge. 

I  will  explain  how  the  phaser  bank  works  and  what  the  controls 
do.  After  I  have  given  you  this  information,  there  will  be  a 
short  quiz  on  the  important  points.  You  must  answer  all  the 
questions  correctly  before  we  can  go  on  to  the  actual  training. 
If  you  answer  a  question  incorrectly,  I  will  present  the 
information  to  you  again,  and  then  you  will  do  the  quiz  again. 
We'll  repeat  this  until  you  can  answer  all  the  questions 
correctly.  You  will  see  that  there  is  a  diagram  above  the  control 
panel.  This  diagram  shows  how  the  phaser  control  system  works. 

The  arrows  on  the  diagram  show  how  power  flows  through  the 
system.  Starting  on  the  lower  left  of  the  diagram,  you  can  see 
that  power  is  drawn  from  the  shipboard  circuits.  This  power  is 
channelled  to  the  energy  booster  (EB),  and  from  there  it  flows  to 
the  two  accumulators  (MA  and  SA).  Power  is  discharged  from  the 
accumulators  to  the  phaser  bank  (PB)  when  the  phasers  are  fired. 
During  training,  the  action  of  the  phaser  system  is  simulated,  so 
that  power  does  not  actually  flow  through  the  system.  This  is 
done  both  to  prevent  accidents  caused  by  phaser  fire,  and  to 
protect  the  delicate  components  of  the  system  from  overload. 

Because  the  components  of  the  system  can  be  damaged,  you  will 
learn  some  procedures  to  be  performed  if  a  component  malfunctions. 
In  order  for  you  to  use  these  malfunction  procedures  correctly  and 
effectively,  it  is  extremely  important  that  you  understand  what 
each  component  does  and  how  the  controls  work.  The  energy  booster 
is  an  essential  part  of  the  system.  Power  drawn  from  the  ship 
cannot  be  used  to  fire  the  phasers  because  it  is  not  at  a  high 
enough  level. 

The  energy  booster  takes  in  power  from  the  ship  and  boosts  it 
to  the  level  necessary  to  fire  the  phasers.  Power  that  has  been 
boosted  by  the  energy  booster  is  fed  into  the  two  accumulators. 
Both  accumulators  store  large  amounts  of  power  ready  to  be 
discharged  to  the  phaser  bank  whenever  the  phasers  are  fired. 
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Because  the  accumulators  handle  such  large  amounts  of  power,  if 
they  are  used  continuously  they  are  liable  to  overload  and  burn 
out.  To  prevent  continuous  use  of  one  accumulator,  this  system 
has  two:  the  main  accumulator  (MA)  and  the  secondary  accumulator 
(SA)  . 

Both  the  main  and  the  secondary  accumulators  store  power 
received  from  the  energy  booster,  and  either  one  can  be  used  to 
discharge  power  to  the  phaser  bank  in  order  to  fire  the  phasers. 
When  one  of  the  phaser  firing  buttons  is  pressed,  the  energy  flows 
from  one  of  the  accumulators  to  the  phaser  bank  (PB). 

The  phaser  bank  can  receive  stored  energy  from  either  the  main 
or  the  secondary  accumulator.  Which  accumulator  will  send  its 
energy  to  the  phaser  bank  is  controlled  by  the  energy  source 
selector  (ES).  Now  that  you  have  seen  what  each  component  does, 
we  will  learn  how  the  controls  on  the  panel  control  the  operation 
of  the  components. 

The  power  coming  in  from  the  shipboard  circuits  is  controlled 
by  the  ship's  power  switch  (SP).  When  this  switch  is  off,  no 
power  is  being  drawn  from  the  ship.  When  the  switch  is  turned  on, 
power  is  drawn  from  the  ship  into  the  energy  booster.  The  boosted 
power  is  then  fed  into  the  accumulators.  The  accumulator  whose 
energy  will  be  discharged  to  the  phaser  banks  is  selected  by  the 
energy  source  selector  (ES).  While  the  ES  selector  is  set  to 
neutral  (N),  no  energy  can  be  discharged  from  either  accumulator 
to  the  phaser  bank. 

When  the  ES  selector  is  set  to  MA  the  power  can  be  drawn  from 
the  main  accumulator.  When  the  ES  selector  is  set  to  SA ,  then 
power  can  be  drawn  from  the  secondary  accumulator.  The  actual 
discharge  of  energy  from  the  selected  accumulator  to  the  phaser 
bank  is  controlled  by  the  firing  buttons,  which  allow  energy  to 
flow  from  the  accumulator  to  the  phaser  bank.  When  the  main 
accumulator  has  been  selected,  the  phasers  are  fired  using  the 
fire  main  (FM)  button.  When  the  secondary  accumulator  has  been 
selected,  then  the  fire  secondary  (FS)  button  fires  the  phasers. 

As  the  diagram  shows,  each  accumulator  has  its  own  firing 
button.  Thus,  if  the  secondary  accumulator  has  been  selected  with 
the  ES  selector,  then  the  phasers  can  only  be  fired  with  the  FS 
button.  If  the  main  accumulator  was  selected,  only  the  FM  button 
will  fire  the  phasers.  Finally,  the  control  panel  is  provided 
with  four  indicator  lights.  Each  indicator  is  attached  to  a 
particulc.*-  component  in  the  system.  The  indicator  will  only  light 
if  the  component  that  it  is  attached  to  is  both  working  and 
receiving  energy. 
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The  SP  indicator  will  light  if  the  phaser  system  is  receiving 
power  from  the  ship.  Thus  the  SP  indicator  will  light  wnen  you 
turn  on  the  SP  switch.  The  energy  booster  (FB'i  indicator  will 
light  if  the  energy  booster  is  receiving  power  from  the  ship,  and 
operating  correctly  and  putting  out  the  boosted  energy. 

The  main  accumulator  (MA)  indicator  will  light  if  the  main 
accumulator  is  receiving  power  from  the  energy  booster,  and  the 
main  accumulator  is  working  and  storing  energy. 

Note  that  there  is  no  indicator  for  the  secondary  accumulator. 
Lastly,  the  phaser  firing  (PF)  indicator  will  light  if  the  phasers 
are  firing.  The  phasers  can  only  fire  if  they  are  both  working 
and  getting  power  from  the  selected  accumulator.  Because  the 
phasers  fire  in  pulses,  the  PF  indicator  will  flash  while  they  are 
firing . 

HI 

You  have  now  learned  two  procedures  for  firing  the  phasers: 
Procedure  1  for  firing  using  the  main  accumulator,  and  Procedure  2 
for  firing  using  the  secondary  accumulator. 

You  will  now  go  on  to  learn  what  you  should  do  if  the  phasers 
do  not  fire  when  you  press  the  appropriate  firing  button.  You 
will  be  taught  procedures  to  follow  in  the  same  way  as  you  were 
taught  the  normal  procedures.  If  the  phasers  do  not  fire  when  you 
press  the  appropriate  firing  button,  it  means  that  one  or  more  of 
the  components  of  the  phaser  control  system  has  malfunctioned.  In 
order  to  determine  the  correct  procedure,  you  must  decide  which 
components  have  failed.  The  indicators  will  provide  you  with 
valuable  clues.  For  example,  you  know  that  the  EB  indicator  will 
light  up  if  the  energy  booster  is  both  working  and  receiving 
power.  If  you  are  supplying  energy,  and  the  indicator  does  not 
light,  you  can  deduce  that  the  energy  booster  is  not  working.  If 
the  only  malfunction  is  in  one  of  the  accumulators,  then  an 
obvious  thing  to  do  is  to  try  firing  from  the  other  accumulator. 
If,  however,  either  the  phaser  ban.<s  or  the  energy  booster  are 
malfunctioning,  then  you  cannot  fire  the  phaser,  no  matter  what 
you  try. 

As  you  work  through  the  malfunction  procedures,  you  will  find 
that  whenever  the  problem  appears  to  be  with  an  accumulator,  you 
will  try  to  fire  using  the  other  accumulator.  if,  however,  the 
problem  is  not  with  the  accumulators,  or  if  there  is  more  than  one 
malfunction,  you  will  be  taught  to  tap  an  "E"  on  the  keyboard  and 
then  power  down  the  phaser  system.  When  you  tap  an  "E" ,  you  are 
informing  Engineering  that  you  have  a  major  malfunction  in  the 
phaser  system.  After  you  have  set  the  ES  selector  to  N  and  turned 
off  ship's  power  with  the  SP  switch,  you  will  still  tap  "F"  for 
finished  so  that  the  engineers  know  that  it  is  safe  to  start 
working  on  the  problem. 
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APPENDIX  B 

Test  Questions  on  Device  Model 
Where  does  the  energy  booster  get  its  power  from? 

(1)  from  the  accumulators. 

(2)  from  the  shipboard  power  circuits. 

(3)  from  its  own  special  power  supply. 

Where  does  the  main  accumulator  get  its  energy  from? 

(1)  from  the  energy  booster. 

(2)  directly  from  the  shipboard  circuits. 

(3)  from  its  own  special  power  supply. 

Where  does  the  secondary  accumulator  get  its  energy  from? 

(1)  directly  from  the  shipboard  circuits. 

(2)  from  the  main  accumulator. 

(3)  from  the  energy  booster. 

Where  does  the  phaser  bank  get  its  energy  from? 

(1)  from  either  one  of  the  two  accumulators. 

(2)  from  the  main  accumulator  only. 

(3)  directly  from  the  energy  booster. 

Whet  is  the  3P  switch  for? 

(1)  It  controls  which  accumulator  will  be  used  for  firing. 

(2)  It  controls  whether  the  phaser  system  can  draw  power 

from  the  ship. 

(3)  It  fires  the  phasers. 

What  does  the  ES  selector  do? 

(1)  It  selects  which  accumulator  the  energy  booster  will 
send  energy  to. 

(2)  It  selects  whether  energy  will  be  received  from  the 
energy  booster  or  not. 

(3)  It  selects  which  accumulator  will  be  used  for  firing  the 
phaser s . 

Assume  that  the  phaser  control  system  is  in  full  working  order  , 
that  power  is  being  supplied  to  the  system,  and  that  the  ES 
selector  is  set  to  MA. 

Now,  what  will  happen  if  the  FM  button  is  pressed? 

(1)  The  main  accumulator  will  discharge  energy  to  the  phaser  banks 
and  the  phasers  will  fire. 

(2)  The  phaser  bank  will  receive  energy  from  the  secondary 
accumulator  and  the  phasers  will  fire. 

(3)  The  phaser  bank  will  receive  energy  directly  from  the  energy 
booster  and  the  phasers  will  fire. 

Assume  that  the  phaser  control  system  is  in  full  working  order, 
that  power  is  being  supplied  to  the  system  and  that  the  ES 
selector  is  set  to  MA. 


Now,  what  will  happen  if  the  FS  button  is  pressed? 

(1)  Nothing.  The  ES  selector  must  be  set  to  SA  for  the  phasers 
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to  fire  when  the  FS  button  is  pressed. 

(2)  The  main  accumulator  will  discharge  power  to  the  phaser  banks 
and  the  phasers  will  fire. 

(3)  The  secondary  accumulator  will  discharge  power  to  the  phaser 
banks  and  the  phasers  will  fire. 

What  does  the  SP  indicator  indicate? 

(1)  It  indicates  whether  the  phasers  are  ready  to  fire  or  not. 

(2)  It  indicates  whether  or  not  the  phaser  system  is  receiving 
power  from  the  ship. 

(3)  It  indicates  whether  the  shipboard  power  circuits  are 
supplying  full  power  or  half  power. 

What  does  it  mean  if  the  EB  indicator  is  on? 

(1)  It  means  that  the  energy  booster  is  receiving  half  power  from 
the  shipboard  power  circuits. 

(2)  It  means  only  that  the  energy  booster  is  receiving  power  from 
the  shipboard  power  circuits. 

(3)  It  means  that  the  energy  booster  is  both  receiving  power  from 
the  shipboard  circuits  and  is  functioning  properly. 

What  does  it  mean  if  the  MA  indicator  is  on? 

(1)  It  means  that  the  main  accumulator  is  discharging  power  to 
the  phasers. 

(2)  It  means  that  the  main  accumlator  is  working  and  is  receiving 
energy  from  the  energy  booster. 

(3)  It  means  that  both  accumlators  are  receiving  energy  and 
working  properly. 

What  can  you  tell  from  the  PF  indicator? 

(1)  If  it  flashes  it  means  that  the  phasers  are  ready  to  fire. 

(2)  If  it  flashes  it  means  that  the  phasers  are  not  working. 

(3)  If  it  flashes  it  means  that  the  phasers  are  firing. 
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Bolt  Beranek  4  Newman 
53  Nouiton  Street 
Cambridge,  MA  02138 

1  Dr.  Alinda  Friedman 

Department  of  Psychology 
University  of  Alberta 
Edmonton,  Alberta 
CANADA  T6G  2E9 

1  Dr.  Michael  Genesereth 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94305 

1  Dr.  Don  Gentner 

Center  for  Human  Information  Processing 
University  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  Dr.  Dedre  Gentner 
Bolt  Beranek  &  Newman 
10  Moulton  St. 

Cambridge,  MA  02138 

1  Dr,  Robert  Glaser 

Learning  Research  6.  Development  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
PITTSBURGH,  PA  15260 

1  Dr.  Marvin  D.  Glock 
217  Stone  Hall 
Cornell  University 
Ithaca,  NY  14853 

1  Dr.  Josph  Goguen 
SRI  International 
333  Ravenswood  Avenue 
Menlo  Park,  CA  94025 

1  Dr.  Daniel  Gopher 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 


1  Dr.  Rert  Green 

Johns  Hopkins  University 
Department  of  Psychology 
Charles  6  34th  Street 
Baltimore,  MD  21218 

1  DR.  JAMES  G.  GREEN 3 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O’HARA  STREET 
PITTSBURGH,  PA  25213 

1  Dr.  Barbara  Hayes-Roth 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  95305 

l  Dr.  Frederick  Hayes-Roth 
Teknowledge 
525  University  Ave. 

Palo  Alto.  CA  94301 

1  Glenda  Greenwald,  Ed. 

Human  Intelligence  Newsletter 
P.  0.  Box  1163 
Birmingham,  Ml  48012 

1  Dr.  Earl  Hunt 

Dept,  of  Psychology 
University  of  Washington 
Seattle,  WA  98105 

1  DR.  KAY  1NABA 
21116  VANOWEN  ST 
CANOGA  PARK,  CA  91303 

1  Dr.  Marcel  Just 

Department  of  Psychology 
Carnegie-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Scott  Kelso 

Haskins  Laboratories,  Inc 
270  Crown  Street 
New  Haven,  CT  06510 

l  Dt.  Walter  Kintsch 

Department  of  Psychology 
University  of  Color a lo 
Boulder,  CO  80302 
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Private  Sector 

1  Dr.  Stephen  Kosslyn 

Department  of  Psychology 
The  Johns  Hopkins  University 
Baltimore,  MD  21218 

1  Dr.  Pat  Langley 

The  Robotics  Institute 
C traegie-Hellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Marcy  Lansman 

The  L.  L.  Thurstone  Psychometric 
Laboratory 

University  of  North  Carolina 
Davie  Hall  0134 
Chapel  Hill,  NC  27514 

1  Dr.  Jill  Larkin 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Alan  Lesgold 
Learning  R&D  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15260 

1  Dr.  Jim  Levin 

University  of  California 
at  San  Diego 

Laboratory  fof  Conparative 
Human  Cognition  -  D003A 
La  Jolla,  CA  92093 

1  Dr.  Michael  Levine 

Department  of  Educational  Psychology 
210  Education  Bldg. 

University  of  Illinois 
Champaign,  IL  61801 

1  Dr.  Marcia  C.  Linn 

University  of  California 

Director,  Adolescent  Reasoning  Project 

Ttrkoley,  CA  94720 

1  Dr.  Jay  McClelland 

Department  of  Psychology 
MIT 

Cambridge,  MA  02139 


Private  Sector 

1  Dr.  James  R.  Miller 

Compute r*TUoaghl  Corporation 
1721  West  Plano  Highway 
Plano,  TX  75075 

1  Dr.  Mark  Miller 

Computer  Thought  Corporation 
1721  West  Plane  Parkway 
Plano,  TX  75075 

1  Dr.  Tom  Moran 
Xerox  PARC 

3333  Coyote  Hill  Road 
Palo  Alto,  CA  94304 

1  Dr.  Allen  Munro 

Behavioral  Technology  Laboratories 
1845  Elena  Ave. ,  Fourth  Floor 
Redondo  Beach,  CA  90277 

1  Dr.  Donald  A  Norman 
Cognitive  Science,  C-015 
Univ.  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  Committee  on  Human  Factors 
JH  811 

2101  Constitution  Ave.  JM 
Washington,  DC  20418 

1  Dr.  Jesse  Orlansky 

Institute  for  Defense  Analyses 
1801  N.  Beauregard  St. 

Alexandria,  VA  22311 

1  Dr.  Seymour  A.  Papert 

Massachusetts  Institute  of  Technology 
Artificial  Intelligence  Lab 
545  Technology  Square 
Cambridge,  MA  02139 

1  Dr.  Nancy  Pennington 
University  of  Chicago 
Graduate  School  of  Business 
HOi  E.  58th  St. 

Chicago,  IL  60637 

1  Mr.  L.  Potrullo 
3695  N.  Nelson  St . 

ARLINGTON,  VA  22207 
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Private  Sector 

1  DR.  PETER  POLSON 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  COLORADO 
BOULDER,  CO  80309 

1  Dr.  Fred  Reif 
Physics  Departraent 
University  of  California 
Berkeley,  CA  94720 

1  Dr.  Lauren  Resnlck 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  1521 

1  Dr.  Jeff  Richardson 

Denver  Research  Institute 
University  of  Denver 
Denver,  CO  80208 

1  Mary  S.  Riley 

Program  in  Cognitive  Science 
Center  for  Human  Information  Processing 
University  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  Dr.  Andrew  M.  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  NW 
Washington,  DC  20007 

!  Or.  Ernst  2.  Rothkcof 
Bell  Laboratories 
Murray  Hill,  NJ  07974 

1  Dr.  William  3.  Rouse 

Georgia  Institute  of  Technology 
School  of  Industrial  5  Systems 
Engineering 
Atlanta,  GA  30332 

1  Dr.  David  Ruraelhart 

Center  for  Human  Information  Processing 
’Jnlv.  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  Dr.  Michael  J.  Saraet 
Perceptronics ,  Inc 
6271  Varlel  Avenue 
Woodland  Hills,  CA  91364 


Private  Sector 

1  Dr.  Arthur  Samuel 
Yale  University 
Department  of  Psychology 
Box  11A,  Yale  Station 
New  Haven,  CT  06520 

1  Dr.  Roger  Schank 
Yale  University 
Department  of  Computer  Science 
P.0.  Box  2158 
New  Haven.  CT  06520 

i  Dr.  Walter  Schneider 
Psychology  Departraent 
603  E.  Daniel 
Champaign,  IT.  61820 

1  Br  -  Alan  &cbeerrftrfd 

Mathematics  and  Education 
The  University  of  Rochester 
Rochester,  NY  14627 

1  DR.  ROBERT  J.  SEIDEL 

INSTRUCTIONAL  TECHNOLOGY  GROUP 
HUMRRO 

300  N.  WASHINGTON  ST. 

ALEXANDRIA,  VA  22314 

1  Mr.  Colin  Sheppard 

Applied  Psychology  Unit 
Admiralty  Marine  Technology  EsL. 
Teddington,  Middlesex 
United  Kingdom 

1  Dr.  H.  Wallace  Sinaiko 
Program  Director 

Manpower  Research  nni  Advisory  Services 
Smithsonian  Institution 

301  North  Pitt  Street 
Alexandria,  VA  22314 

1  Dr.  Edward  E.  Smith 

Bolt  Beranek  5  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 

l  Dr.  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 
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1  Dr.  Eliott  Soloway 
Yale  University 

Department  of  Computer  Science 

P.0.  Box  2158 

New  Haven,  CT  06520 

1  Dr.  Kathryn  T.  Spoehr 
Psychology  Department 
Brown  University 
Providence,  RI  02912 

I  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  114,  Yale  Station 
New  Haven,  CT  06520 

1  Dr.  Albert  Steven6 

Bolt  Beranek  &  Newman,  Inc. 

10  Moulton  St. 

Cambridge,  MA  02238 

1  David  E,  Stone,  Ph.D. 

Mazeltine  Corporation 
7680  Old  Sprtnghouse  Road 
McLean,  VA  22102 

1  DR.  PATRICK  SUPPES 

INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  94305 

1  Dr.  Kikvni  Tatsuoku 

loaputer  Based  Education  Research  Lab 
252  Engineering  Resaarch  Laboratory 
Urbans,  IL  01801 

1  Dr.  Maurice  Tatsuoka 
220  Education  Rldg 
1310  S.  Sixth  St. 

Champaign,  IL  61820 

l  Dr.  Perry  W.  Thorndyke 
Perceptronics,  Inc. 

545  Mlddlefleld  Road,  Suite  140 
Menlo  Park,  CA  94025 

1  Or.  Douglas  To\/ne 

Univ.  of  So.  California 
Behavioral  Technology  Labs 
1345  S.  Elena  Ave. 

Redondo  Beach,  CA  90277 


Private  Sector 

1  Dr.  Kurt  Van  Lehn 
Zerox  PARC 

3333  Coyote  Hill  Road 
Palo  Alto,  CA  94304 

1  DR.  GERSHON  WELTMAN 
PERCEPTRONICS  INC. 

6271  VARIEL  AVE. 

WOODLAND  HILLS,  CA  91367 

1  Dr.  Keith  T.  Wescourt 
Perceptronics ,  Inc. 

545  Middlefield  Road,  Suite  140 
Menlo  Park,  CA  94025 

1  William  B.  Whitten 
Bell  Laboratories 
2D-610 

Holradel ,  NJ  07733 

1  Dr.  Mike  Williams 
Zerox  PARC 

3333  Coyote  Hill  Road 
Palo  Alto,  CA  94304 


